Deployments

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
  <siteinfo>
    <sitename>Wikitech</sitename>
    <dbname>labswiki</dbname>
    <base>https://wikitech.wikimedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.48.0-wmf.3</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikitech</namespace>
      <namespace key="5" case="first-letter">Wikitech talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="110" case="first-letter">Obsolete</namespace>
      <namespace key="111" case="first-letter">Obsolete talk</namespace>
      <namespace key="112" case="first-letter">OfficeIT</namespace>
      <namespace key="113" case="first-letter">OfficeIT talk</namespace>
      <namespace key="116" case="first-letter">Tool</namespace>
      <namespace key="117" case="first-letter">Tool talk</namespace>
      <namespace key="498" case="first-letter">Nova Resource</namespace>
      <namespace key="499" case="first-letter">Nova Resource Talk</namespace>
      <namespace key="666" case="first-letter">Heira</namespace>
      <namespace key="667" case="first-letter">Heira Talk</namespace>
      <namespace key="710" case="first-letter">TimedText</namespace>
      <namespace key="711" case="first-letter">TimedText talk</namespace>
      <namespace key="828" case="first-letter">Module</namespace>
      <namespace key="829" case="first-letter">Module talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Deployments</title>
    <ns>0</ns>
    <id>4108</id>
    <revision>
      <id>2479015</id>
      <parentid>2478954</parentid>
      <timestamp>2026-11-08T05:41:09Z</timestamp>
      <contributor>
        <username>ScheduleDeploymentBot</username>
        <id>37566</id>
      </contributor>
      <comment>Add [[gerrit:1358854]] to Monday, November 09 UTC late backport window</comment>
      <origin>2479015</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="44694" sha1="fipbx89wo0rf93f24cwm9392w58rc6p" xml:space="preserve">{{Navigation MediaWiki deployment}}
This page tracks '''upcoming''' '''deployments''' of software to the [[:m:Special:SiteMatrix|Wikimedia Foundation servers]].

== Getting started ==
Ensure you joined the {{irc|wikimedia-operations}} IRC channel as all deployment-related communications happen there.
If you need help, contact [[:mw:Wikimedia Release Engineering Team|Release Engineering]] on IRC at {{irc|wikimedia-releng}}; and ping Tyler (&lt;code&gt;thcipriani&lt;/code&gt;).

* '''MediaWiki is deployed weekly''' through the [[/Train|Deployment Train]]. Other services follow their own schedule.
* '''Times are pinned to San Francisco''', thus the UTC time changes in March and November per [[:en:Daylight saving time in the United States|DST]].
* '''Prefer regular [[Backport windows]]''' over adding new windows. To request deployment of a config change or backport, add your username and Gerrit URL to one of the backport windows on this page. You must be online in #wikimedia-operations on IRC during your deployment and install [[WikimediaDebug]] ahead of time. The #wikimedia-operations channel requires you to [[:m:IRC/Instructions#Register your nickname, identify, and enforce|register your nickname]] before you can join.
** You can use the '''backport scheduling tool''' to more easily edit this page: &lt;div style=&quot;text-align: center; margin: 1em 0&quot;&gt;{{Clickable button 2|:toollabs:schedule-deployment|Schedule a backport|class=mw-ui-progressive}}&lt;/div&gt;
* Tasks that meet [[/Inclusion criteria|Inclusion criteria]] '''require their own windows''', which includes long-running tasks. '''Schedule more time''' than you think you need to account for delays and set backs, we recommend one hour for most tasks.
**To create or modify a recurring deploy window, send a patchset to [[:gitlab:repos/releng/release/-/blob/main/make-deployment-calendar/deployments-calendar.yaml|deployments-calendar.yaml file]] in &lt;code&gt;repos/releng/release.git&lt;/code&gt;.
**To create an one-off window, simply edit this page accordingly
** '''Announce''' changes to the [[mail:ops|ops mailing list]] ahead of time if you anticipate or are uncertain about noticeable impacts to database load, HTTP caching, or the introduction of new cookies.
** '''Announce''' deployments of major features to the community via [[:m:Tech/News/Next|Tech News]] and/or via other [[:mw:Wikimedia_Product_Guidance/Communication_channels|Product communication channels]].
* '''Something went wrong?''' See [[Incident response]]. Is there a user-impacting problem? Communicate in the {{irc|wikimedia-operations}} IRC channel. If there is a Phabricator task, ensure [[:phab:tag/wikimedia-incident/|#Wikimedia-Incident]] is tagged, and consider setting the [[:mw:Phabricator/Project_management#Priority_levels|Unbreak Now]] priority.
__TOC__
{{anchor|Next Week|Near Term|Near term|Near-term}}{{clear}}
[[Category:Deployment]]
{{Note|content=Subscribe in Google Calendar via &lt;code&gt;wikimedia.org_rudis09ii2mm5fk4hgdjeh1u64@group.calendar.google.com&lt;/code&gt;.&lt;br&gt;This may not include one-off windows. '''If there are differences, then the wiki page is canonical and correct'''.}}

==Week of November 09==


==={{Deployment_day|date=2026-11-08}}===
{{Deployment calendar event card
    |when=2026-11-08 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
==={{Deployment_day|date=2026-11-09}}===
{{Deployment calendar event card
    |when=2026-11-09 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-09 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-09 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-09 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|jan_drewniak|Jan Drewniak}}
{{deploy|type=config|gerrit=1364782|title=Enable ReadingLists custom lists on all Wikipedias.|status=}} - {{phabricator|T441372}}
{{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-09 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-09 08:30 SF
    |length=0.5
    |window=Wikimedia Portals Update
    |who={{ircnick|jan_drewniak|Jan Drewniak}}
    |what=Weekly window for the portals page: https://www.wikipedia.org/
}}
{{Deployment calendar event card
    |when=2026-11-09 09:00 SF
    |length=1
    |window=Release Engineering Clinic Duty
    |who={{ircnick|thcipriani|Tyler}}
    |what=[[mw:Wikimedia_Release_Engineering_Team/Clinic_duty|RelEng Clinic Duty]] is a rotating position who handles incoming requests for Release Engineering (RelEng).
}}
{{Deployment calendar event card
    |when=2026-11-09 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-09 10:00 SF
    |length=0.5
    |window=Wikidata Query Service weekly deploy
    |who={{ircnick|ryankemper|Ryan}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-11-09 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|EricGardner|EricGardner}}
{{deploy|type=config|gerrit=1358854|title=Enable the ShareCard feature on all Wikipedias (Minerva only)|status=}} - {{phabricator|T439162}}
{{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-09 14:00 SF
    |length=2
    |window=Weekly Security deployment window
    |who={{ircnick|alexsanford|Alex}}, {{ircnick|Reedy|Sam}}, {{ircnick|sbassett|Scott}}, {{ircnick|Maryum|Maryum}}, {{ircnick|manfredi|Manfredi}}
    |what=Held deployment window for Security-team related deploys.
}}
{{Deployment calendar event card
    |when=2026-11-09 16:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-09 19:00 SF
    |length=1
    |window=Automatic branching of MediaWiki, extensions, skins, and vendor – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Branch &lt;code&gt;wmf/1.48.0-wmf.4&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-11-09 20:00 SF
    |length=1
    |window=Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/1.48.0-wmf.4&lt;/code&gt; to testwikis
}}
{{Deployment calendar event card
    |when=2026-11-09 21:00 SF
    |length=1
    |window=Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version)
    |who=N/A
    |what=Runs &lt;code&gt;scap clean auto&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-11-09 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-09 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}
==={{Deployment_day|date=2026-11-10}}===
{{Deployment calendar event card
    |when=2026-11-10 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-10 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0+Utc-7 Version
    |who={{ircnick|andre|Andre}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.3-&gt;1.48.0-wmf.4|1.48.0-wmf.3|1.48.0-wmf.3}}
* group0 to [[mw:MediaWiki_1.48/wmf.4|1.48.0-wmf.4]]
* '''Blockers: {{phabricator|T442222}}'''
}}
{{Deployment calendar event card
    |when=2026-11-10 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-10 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-10 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-11-10 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-10 07:00 SF
    |length=0.5
    |window=Test Kitchen UI Deployment Window
    |who=Experimentation Platform Team
    |what=Deployment of Test Kitchen UI (fka MPIC)
}}
{{Deployment calendar event card
    |when=2026-11-10 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-10 08:00 SF
    |length=1
    |window=SRE Collaboration Services office hours
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=Services including Gerrit, Phorge (Phabricator), GitLab
}}
{{Deployment calendar event card
    |when=2026-11-10 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-11-10 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-10 11:00 SF
    |length=2
    |window=MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot)
    |who={{ircnick|andre|Andre}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.3-&gt;1.48.0-wmf.4|1.48.0-wmf.3|1.48.0-wmf.3}}
* group0 to [[mw:MediaWiki_1.48/wmf.4|1.48.0-wmf.4]]
* '''Blockers: {{phabricator|T442222}}'''
}}
{{Deployment calendar event card
    |when=2026-11-10 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|sbassett|sbassett}}
{{deploy|type=config|gerrit=1363494|title=Enable DBSC report-only mode on all Wikimedia wikis|status=}} - {{phabricator|T393212}}
{{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-10 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-10 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-11-11}}===
{{Deployment calendar event card
    |when=2026-11-11 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-11 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0+Utc-7 Version
    |who={{ircnick|andre|Andre}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.4|1.48.0-wmf.3-&gt;1.48.0-wmf.4|1.48.0-wmf.3}}
* group1 to [[mw:MediaWiki_1.48/wmf.4|1.48.0-wmf.4]]
* '''Blockers: {{phabricator|T442222}}'''
}}
{{Deployment calendar event card
    |when=2026-11-11 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-11 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-11 04:00 SF
    |length=1
    |window=[[mw:Services|Services]] – [[Citoid]] / [[Zotero]]  
    |who=Marielle ({{ircnick|mvolz}})
    |what=See [[mw:Citoid|Citoid]]
}}
{{Deployment calendar event card
    |when=2026-11-11 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-11 07:00 SF
    |length=1
    |window=Wikifunctions Services UTC Afternoon
    |who=Abstract Wikipedia team (Africa, Europe, Eastern Americas)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-11-11 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-11 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-11 11:00 SF
    |length=2
    |window=MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot)
    |who={{ircnick|andre|Andre}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.4|1.48.0-wmf.3-&gt;1.48.0-wmf.4|1.48.0-wmf.3}}
* group1 to [[mw:MediaWiki_1.48/wmf.4|1.48.0-wmf.4]]
* '''Blockers: {{phabricator|T442222}}'''
}}
{{Deployment calendar event card
    |when=2026-11-11 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-11 14:00 SF
    |length=1
    |window=Wikifunctions Services UTC Late
    |who=Abstract Wikipedia team (North and South America)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-11-11 15:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-11 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-11 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}
==={{Deployment_day|date=2026-11-12}}===
{{Deployment calendar event card
    |when=2026-11-12 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-12 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0+Utc-7 Version
    |who={{ircnick|andre|Andre}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.4|1.48.0-wmf.4|1.48.0-wmf.3-&gt;1.48.0-wmf.4}}
* group2 to [[mw:MediaWiki_1.48/wmf.4|1.48.0-wmf.4]]
* '''Blockers: {{phabricator|T442222}}'''
}}
{{Deployment calendar event card
    |when=2026-11-12 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-12 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-12 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-11-12 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-12 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-12 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-11-12 10:00 SF
    |length=1
    |window=Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker)
    |who={{ircnick|bd808}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-11-12 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-12 11:00 SF
    |length=2
    |window=MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot)
    |who={{ircnick|andre|Andre}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.4|1.48.0-wmf.4|1.48.0-wmf.3-&gt;1.48.0-wmf.4}}
* group2 to [[mw:MediaWiki_1.48/wmf.4|1.48.0-wmf.4]]
* '''Blockers: {{phabricator|T442222}}'''
}}
{{Deployment calendar event card
    |when=2026-11-12 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-12 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-12 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-11-13}}===
{{Deployment calendar event card
    |when=2026-11-13 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
{{Deployment calendar event card
    |when=2026-11-13 04:00 SF
    |length=0.5
    |window=GitLab version upgrades
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=GitLab version upgrades
}}
==={{Deployment_day|date=2026-11-14}}===
{{Deployment calendar event card
    |when=2026-11-14 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}

==Week of November 16==


==={{Deployment_day|date=2026-11-15}}===
{{Deployment calendar event card
    |when=2026-11-15 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
==={{Deployment_day|date=2026-11-16}}===
{{Deployment calendar event card
    |when=2026-11-16 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-16 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-16 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-16 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-16 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-16 08:30 SF
    |length=0.5
    |window=Wikimedia Portals Update
    |who={{ircnick|jan_drewniak|Jan Drewniak}}
    |what=Weekly window for the portals page: https://www.wikipedia.org/
}}
{{Deployment calendar event card
    |when=2026-11-16 09:00 SF
    |length=1
    |window=Release Engineering Clinic Duty
    |who={{ircnick|thcipriani|Tyler}}
    |what=[[mw:Wikimedia_Release_Engineering_Team/Clinic_duty|RelEng Clinic Duty]] is a rotating position who handles incoming requests for Release Engineering (RelEng).
}}
{{Deployment calendar event card
    |when=2026-11-16 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-16 10:00 SF
    |length=0.5
    |window=Wikidata Query Service weekly deploy
    |who={{ircnick|ryankemper|Ryan}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-11-16 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-16 14:00 SF
    |length=2
    |window=Weekly Security deployment window
    |who={{ircnick|alexsanford|Alex}}, {{ircnick|Reedy|Sam}}, {{ircnick|sbassett|Scott}}, {{ircnick|Maryum|Maryum}}, {{ircnick|manfredi|Manfredi}}
    |what=Held deployment window for Security-team related deploys.
}}
{{Deployment calendar event card
    |when=2026-11-16 16:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-16 19:00 SF
    |length=1
    |window=Automatic branching of MediaWiki, extensions, skins, and vendor – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Branch &lt;code&gt;wmf/1.48.0-wmf.5&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-11-16 20:00 SF
    |length=1
    |window=Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/1.48.0-wmf.5&lt;/code&gt; to testwikis
}}
{{Deployment calendar event card
    |when=2026-11-16 21:00 SF
    |length=1
    |window=Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version)
    |who=N/A
    |what=Runs &lt;code&gt;scap clean auto&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-11-16 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-16 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}
==={{Deployment_day|date=2026-11-17}}===
{{Deployment calendar event card
    |when=2026-11-17 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-17 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-17 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-17 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-11-17 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-17 07:00 SF
    |length=0.5
    |window=Test Kitchen UI Deployment Window
    |who=Experimentation Platform Team
    |what=Deployment of Test Kitchen UI (fka MPIC)
}}
{{Deployment calendar event card
    |when=2026-11-17 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-17 08:00 SF
    |length=1
    |window=SRE Collaboration Services office hours
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=Services including Gerrit, Phorge (Phabricator), GitLab
}}
{{Deployment calendar event card
    |when=2026-11-17 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-11-17 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-17 11:00 SF
    |length=2
    |window=MediaWiki train - Utc-7 Version
    |who={{ircnick|jeena|Jeena}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.4-&gt;1.48.0-wmf.5|1.48.0-wmf.4|1.48.0-wmf.4}}
* group0 to [[mw:MediaWiki_1.48/wmf.5|1.48.0-wmf.5]]
* '''Blockers: {{phabricator|T442223}}'''
}}
{{Deployment calendar event card
    |when=2026-11-17 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-17 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-17 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-11-18}}===
{{Deployment calendar event card
    |when=2026-11-18 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-18 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-18 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-18 04:00 SF
    |length=1
    |window=[[mw:Services|Services]] – [[Citoid]] / [[Zotero]]  
    |who=Marielle ({{ircnick|mvolz}})
    |what=See [[mw:Citoid|Citoid]]
}}
{{Deployment calendar event card
    |when=2026-11-18 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-18 07:00 SF
    |length=1
    |window=Wikifunctions Services UTC Afternoon
    |who=Abstract Wikipedia team (Africa, Europe, Eastern Americas)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-11-18 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-18 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-18 11:00 SF
    |length=2
    |window=MediaWiki train - Utc-7 Version
    |who={{ircnick|jeena|Jeena}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.5|1.48.0-wmf.4-&gt;1.48.0-wmf.5|1.48.0-wmf.4}}
* group1 to [[mw:MediaWiki_1.48/wmf.5|1.48.0-wmf.5]]
* '''Blockers: {{phabricator|T442223}}'''
}}
{{Deployment calendar event card
    |when=2026-11-18 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-18 14:00 SF
    |length=1
    |window=Wikifunctions Services UTC Late
    |who=Abstract Wikipedia team (North and South America)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-11-18 15:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-18 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-18 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}
==={{Deployment_day|date=2026-11-19}}===
{{Deployment calendar event card
    |when=2026-11-19 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}, {{ircnick|chlod|Chlod}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-19 02:00 UTC
    |length=1
    |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/next&lt;/code&gt; to pretrain environment
}}
{{Deployment calendar event card
    |when=2026-11-19 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-19 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-11-19 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-19 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-11-19 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-11-19 10:00 SF
    |length=1
    |window=Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker)
    |who={{ircnick|bd808}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-11-19 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-11-19 11:00 SF
    |length=2
    |window=MediaWiki train - Utc-7 Version
    |who={{ircnick|jeena|Jeena}}, {{ircnick|thcipriani|Tyler}}
    |what=[[mw:MediaWiki 1.48/Roadmap#Schedule for the deployments|1.48 schedule]]
{{DeployOneWeekMini|1.48.0-wmf.5|1.48.0-wmf.5|1.48.0-wmf.4-&gt;1.48.0-wmf.5}}
* group2 to [[mw:MediaWiki_1.48/wmf.5|1.48.0-wmf.5]]
* '''Blockers: {{phabricator|T442223}}'''
}}
{{Deployment calendar event card
    |when=2026-11-19 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-11-19 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-11-19 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-11-20}}===
{{Deployment calendar event card
    |when=2026-11-20 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
{{Deployment calendar event card
    |when=2026-11-20 04:00 SF
    |length=0.5
    |window=GitLab version upgrades
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=GitLab version upgrades
}}
==={{Deployment_day|date=2026-11-21}}===
{{Deployment calendar event card
    |when=2026-11-21 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}</text>
      <sha1>fipbx89wo0rf93f24cwm9392w58rc6p</sha1>
    </revision>
  </page>
  <page>
    <title>Server Admin Log</title>
    <ns>0</ns>
    <id>7919</id>
    <revision>
      <id>2478966</id>
      <parentid>2478956</parentid>
      <timestamp>2026-11-07T12:00:05Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync</comment>
      <origin>2478966</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="319388" sha1="gpo538njlmcvi0cbo0o175v0ejkhd0f" xml:space="preserve">== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>gpo538njlmcvi0cbo0o175v0ejkhd0f</sha1>
    </revision>
    <revision>
      <id>2478967</id>
      <parentid>2478966</parentid>
      <timestamp>2026-11-07T12:00:15Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync</comment>
      <origin>2478967</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="319472" sha1="3cqg2xqt4sc2lrvt3mtlnr74sxq42mi" xml:space="preserve">== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>3cqg2xqt4sc2lrvt3mtlnr74sxq42mi</sha1>
    </revision>
    <revision>
      <id>2478968</id>
      <parentid>2478967</parentid>
      <timestamp>2026-11-07T12:00:20Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2478968</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="319566" sha1="f0r3k64bay0fau4u3c7zndr6kj4v6je" xml:space="preserve">== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>f0r3k64bay0fau4u3c7zndr6kj4v6je</sha1>
    </revision>
    <revision>
      <id>2478969</id>
      <parentid>2478968</parentid>
      <timestamp>2026-11-07T12:00:30Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2478969</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="319659" sha1="7dxa5kvep0jh4ezueui0msvpaxkab3b" xml:space="preserve">== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>7dxa5kvep0jh4ezueui0msvpaxkab3b</sha1>
    </revision>
    <revision>
      <id>2478970</id>
      <parentid>2478969</parentid>
      <timestamp>2026-11-07T12:00:33Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2478970</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="319751" sha1="7ixen88dayfnhm1ohztjdcscxh1lb66" xml:space="preserve">== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>7ixen88dayfnhm1ohztjdcscxh1lb66</sha1>
    </revision>
    <revision>
      <id>2478971</id>
      <parentid>2478970</parentid>
      <timestamp>2026-11-07T12:00:40Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2478971</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="319842" sha1="c1il3plxmay06doablyw53n9q34xs4p" xml:space="preserve">== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>c1il3plxmay06doablyw53n9q34xs4p</sha1>
    </revision>
    <revision>
      <id>2479008</id>
      <parentid>2478971</parentid>
      <timestamp>2026-11-08T00:00:08Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2479008</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="319954" sha1="3ife6ay9qeb79gb4eys3yosxmoi2ajb" xml:space="preserve">== 2026-11-08 ==
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>3ife6ay9qeb79gb4eys3yosxmoi2ajb</sha1>
    </revision>
    <revision>
      <id>2479009</id>
      <parentid>2479008</parentid>
      <timestamp>2026-11-08T00:00:17Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2479009</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="320047" sha1="2v1b6nno2iw7q42rm38e4b9l2u1rtuo" xml:space="preserve">== 2026-11-08 ==
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>2v1b6nno2iw7q42rm38e4b9l2u1rtuo</sha1>
    </revision>
    <revision>
      <id>2479010</id>
      <parentid>2479009</parentid>
      <timestamp>2026-11-08T00:00:20Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2479010</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="320139" sha1="9qhm2jfqd4vse58n7e07xn5gtx2oo0e" xml:space="preserve">== 2026-11-08 ==
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>9qhm2jfqd4vse58n7e07xn5gtx2oo0e</sha1>
    </revision>
    <revision>
      <id>2479011</id>
      <parentid>2479010</parentid>
      <timestamp>2026-11-08T00:00:27Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync</comment>
      <origin>2479011</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="320230" sha1="7ahafrkukdf9g2f0wu7ufybcjxvr2x7" xml:space="preserve">== 2026-11-08 ==
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>7ahafrkukdf9g2f0wu7ufybcjxvr2x7</sha1>
    </revision>
    <revision>
      <id>2479013</id>
      <parentid>2479011</parentid>
      <timestamp>2026-11-08T02:01:05Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image</comment>
      <origin>2479013</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="320313" sha1="7xdl750kvnby473v1mm0rmm5sgbpsw3" xml:space="preserve">== 2026-11-08 ==
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>7xdl750kvnby473v1mm0rmm5sgbpsw3</sha1>
    </revision>
    <revision>
      <id>2479014</id>
      <parentid>2479013</parentid>
      <timestamp>2026-11-08T02:08:37Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)</comment>
      <origin>2479014</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="320417" sha1="2coqls9geqbxf6p1mfbr17bazx0m48p" xml:space="preserve">== 2026-11-08 ==
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-07 ==
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
* 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-06 ==
* 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
* 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
* 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;kubestagemaster2005.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
* 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
* 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
* 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
* 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
* 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
* 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
* 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
* 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
* 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
* 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
* 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
* 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
* 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
* 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
* 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
* 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
* 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
* 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
* 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
* 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
* 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
* 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
* 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
* 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
* 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
* 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
* 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004&quot;
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
* 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004&quot;
* 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
* 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
* 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
* 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
* 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
* 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
* 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
* 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
* 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
* 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
* 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
* 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
* 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
* 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
* 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
* 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
* 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
* 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
* 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
* 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
* 10:42 effie: depool kartotherian from eqiad (ie maps) - [[phab:T443438|T443438]]
* 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]] (duration: 03m 39s)
* 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443357|T443357]]
* 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* [[phab:T442995|T442995]]
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
* 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
* 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
* 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]] (duration: 04m 11s)
* 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
* 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
* 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix [[phab:T443826|T443826]]
* 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
* 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
* 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
* 06:52 XioNoX: push pfw policies - [[phab:T443852|T443852]]
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-05 ==
* 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
* 23:18 maryum: Deployed security fix for [[phab:T443424|T443424]], [[phab:T443467|T443467]], [[phab:T443036|T443036]]
* 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
* 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
* 22:51 sbassett: Deployed security fixes for [[phab:T439878|T439878]], [[phab:T440323|T440323]], [[phab:T440657|T440657]], [[phab:T440656|T440656]], [[phab:T440655|T440655]]
* 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
* 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
* 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
* 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
* 22:31 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s)
* 22:24 egardner@deploy2003: egardner: Continuing with deployment
* 22:23 egardner@deploy2003: egardner: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
* 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:19 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1364473{{!}}ShareCard: Log client errors with the topic &quot;error.sharecard&quot; (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474{{!}}ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3)
* 22:15 egardner@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s)
* 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
* 22:08 egardner@deploy2003: egardner: Continuing with deployment
* 22:07 egardner@deploy2003: egardner: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
* 22:01 egardner@deploy2003: Started scap sync-world: Backport for [[gerrit:1358851{{!}}Add Creative Commons license icons for the share card (T439162)]]
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
* 21:54 mutante: ci / contint1003/contint2003 - if &quot;sudo docker info -f '{{.Driver}}'&quot; says &quot;overlay2&quot; it's the classic way. after restart when it says &quot;overlayfs2&quot; that will be the NEW way! ([[phab:T440668|T440668]])
* 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
* 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
* 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4  ([[phab:T440668|T440668]])
* 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
* 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
* 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add [[phab:T440859|T440859]]
* 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix  # [[phab:T430731|T430731]]
* 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
* 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
* 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s)
* 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
* 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
* 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
* 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
* 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
* 21:14 kemayo@deploy2003: pppery, kemayo: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:08 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1364770{{!}}VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815{{!}}Add new namespace for zh_min_nanwiki (T430731)]]
* 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
* 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
* 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # [[phab:T443295|T443295]]
* 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
* 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former [[phab:T440859|T440859]]
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
* 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
* 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
* 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
* 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
* 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
* 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
* 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
* 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
* 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
* 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
* 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
* 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
* 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
* 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
* 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
* 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
* 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
* 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
* 18:57 dancy@deploy2003: Finished scap sync-world: Testing [[phab:T411063|T411063]] (duration: 44m 09s)
* 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
* 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
* 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
* 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
* 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
* 18:42 dancy@deploy2003: dancy: Continuing with deployment
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
* 18:35 dancy@deploy2003: dancy: Testing [[phab:T411063|T411063]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
* 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
* 18:13 dancy@deploy2003: Started scap sync-world: Testing [[phab:T411063|T411063]]
* 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
* 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
* 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
* 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
* 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
* 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
* 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003&quot;
* 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
* 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
* 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
* 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
* 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 [[phab:T443687|T443687]]
* 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
* 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
* 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
* 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
* 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing ([[phab:T443632|T443632]]) ([[phab:T443640|T443640]])
* 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
* 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
* 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 16:40 moritzm: installing krb5 security updates
* 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]] (duration: 03m 03s)
* 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - [[phab:T439955|T439955]]
* 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 16:35 moritzm: installing krb5 security updates
* 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
* 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2013.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P&lt;nowiki&gt;{&lt;/nowiki&gt;lvs2014.codfw.wmnet&lt;nowiki&gt;}&lt;/nowiki&gt; and A:lvs
* 15:10 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s)
* 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 stran@deploy2003: stran, codenamenoreste: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 14:58 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1362175{{!}}Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399{{!}}Enable Special:SuggestedInvestigations for nlwiki (T443646)]]
* 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s)
* 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
* 14:38 mlitn@deploy2003: mlitn: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:34 mlitn@deploy2003: Started scap sync-world: Backport for [[gerrit:1364301{{!}}Squashed diff to master]] (1.48.0-wmf.3)
* 14:32 Tran: Deployed patch for [[phab:T443598|T443598]]
* 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
* 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
* 14:15 stran@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s)
* 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 14:09 stran@deploy2003: stran: Continuing with deployment
* 14:08 stran@deploy2003: stran: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
* 14:03 stran@deploy2003: Started scap sync-world: Backport for [[gerrit:1364209{{!}}Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]]
* 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
* 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
* 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
* 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
* 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
* 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
* 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
* 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
* 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
* 12:52 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s)
* 12:46 kamila@deploy2003: kamila: Continuing with deployment
* 12:46 kamila@deploy2003: kamila: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
* 12:41 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1364180{{!}}Enroll 5% of client sessions in PHP 8.5 (T441689)]]
* 12:37 moritzm: installing alsa-lib security updates
* 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
* 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
* 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
* 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
* 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:11 moritzm: installing alsa-lib security updates
* 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
* 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
* 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
* 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
* 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
* 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
* 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
* 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
* 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
* 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
* 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
* 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
* 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
* 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
* 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
* 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
* 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
* 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
* 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
* 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
* 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
* 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
* 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
* 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
* 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
* 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
* 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
* 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
* 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
* 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
* 10:32 moritzm: installing libgit2 security updates
* 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
* 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
* 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
* 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
* 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
* 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
* 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
* 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
* 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
* 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
* 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
* 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
* 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
* 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
* 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
* 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
* 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
* 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
* 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
* 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
* 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
* 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
* 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
* 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
* 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
* 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
* 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
* 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
* 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
* 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
* 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s)
* 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo [[phab:T441395|T441395]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004&quot;
* 08:10 dcausse@deploy2003: dcausse: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
* 08:06 dcausse@deploy2003: Started scap sync-world: Backport for [[gerrit:1363978{{!}}Restore the semantic highlighting batch ext]] (1.48.0-wmf.3)
* 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
* 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
* 07:56 moritzm: installing expat security updates
* 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
* 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
* 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
* 07:26 marostegui: Drop thread_reaction tables [[phab:T442289|T442289]]
* 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
* 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
* 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
* 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
* 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
* 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
* 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
* 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
* 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
* 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
* 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
* 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
* 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
* 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
* 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
* 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2020.codfw.wmnet, repooling both afterwards
* 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
* 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1026.eqiad.wmnet, repooling both afterwards
* 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2019.codfw.wmnet, repooling both afterwards
* 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1025.eqiad.wmnet, repooling both afterwards
* 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2018.codfw.wmnet, repooling both afterwards
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
* 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
* 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
* 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
* 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
* 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
* 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
* 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
* 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
* 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
* 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
* 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
* 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
* 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
* 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
* 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
* 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
* 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
* 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
* 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
* 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
* 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
* 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
* 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
* 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
* 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
* 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
* 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
* 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
* 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-04 ==
* 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
* 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
* 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
* 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
* 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
* 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
* 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s)
* 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
* 23:15 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1363460{{!}}UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3)
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
* 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
* 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
* 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
* 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
* 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
* 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
* 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
* 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
* 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
* 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
* 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
* 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
* 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) (duration: 41m 33s)
* 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
* 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
* 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
* 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
* 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
* 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
* 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
* 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
* 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
* 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
* 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php  # [[phab:T443456|T443456]]
* 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
* 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
* 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265{{!}}1.48.0-wmf.1]], [[gerrit:1363263{{!}}1.48.0-wmf.3]])
* 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
* 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
* 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
* 21:22 tsev@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s)
* 21:16 tsev@deploy2003: tsev: Continuing with deployment
* 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
* 21:13 tsev@deploy2003: tsev: Backport for [[gerrit:1363467{{!}}Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:37 mutante: !log is not !logging
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] (duration: 09m 46s)
* 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
* 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1018.eqiad.wmnet, repooling both afterwards
* 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2014.codfw.wmnet, repooling both afterwards
* 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363423{{!}}Redesign 404 pages (T129433)]]
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2013.codfw.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1017.eqiad.wmnet, repooling both afterwards
* 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
* 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
* 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1016.eqiad.wmnet, repooling both afterwards
* 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2012.codfw.wmnet, repooling both afterwards
* 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]]) (duration: 09m 40s)
* 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2011.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1014.eqiad.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns ([[phab:T129433|T129433]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2010.codfw.wmnet, repooling both afterwards
* 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1013.eqiad.wmnet, repooling both afterwards
* 18:35 ladsgroup@deploy2003: Stopping before sync operations
* 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns ([[phab:T129433|T129433]])
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
* 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
* 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
* 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs2007.codfw.wmnet, repooling both afterwards
* 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -&gt; wdqs1012.eqiad.wmnet, repooling both afterwards
* 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s)
* 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 ([[phab:T431398|T431398]])
* 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
* 18:01 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:58 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311062{{!}}Remove Mathoid SVG option from wgMathValidModes (T271001)]]
* 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
* 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
* 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
* 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
* 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia ([[phab:T443632|T443632]])
* 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
* 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
* 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
* 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] (duration: 11m 19s)
* 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
* 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
* 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 16:53 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1356470{{!}}Redesign 503 pages (T129433)]]
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
* 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
* 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T441682|T441682]], seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -&gt; wdqs1011.eqiad.wmnet, repooling both afterwards
* 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
* 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
* 16:06 kamila@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s)
* 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, [[phab:T416563|T416563]]]
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
* 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
* 16:00 kamila@deploy2003: kamila: Continuing with deployment
* 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
* 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
* 15:58 kamila@deploy2003: kamila: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:56 kamila@deploy2003: Started scap sync-world: Backport for [[gerrit:1363270{{!}}Enroll 1% of client sessions in PHP 8.5 (T441689)]]
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004&quot;
* 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
* 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
* 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
* 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
* 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
* 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
* 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
* 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
* 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s)
* 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
* 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
* 15:24 kharlan@deploy2003: kharlan: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:22 kharlan@deploy2003: Started scap sync-world: Backport for [[gerrit:1363260{{!}}AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3)
* 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
* 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
* 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
* 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
* 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s)
* 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
* 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 ([[phab:T441633|T441633]])', diff saved to  and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
* 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
* 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}}) (duration: 01m 46s)
* 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second ({{Gerrit|1363248}})
* 15:05 sbisson@deploy2003: sbisson: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:05 papaul: start maintenance on cr2-eqsin
* 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
* 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
* 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
* 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}}) (duration: 02m 28s)
* 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
* 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
* 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
* 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first ({{Gerrit|1363247}})
* 15:00 sbisson@deploy2003: Started scap sync-world: Backport for [[gerrit:1357358{{!}}ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118{{!}}Load ArticleGuidance on all Wikipedias (T442035)]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix  # [[phab:T440137|T440137]]
* 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary  # [[phab:T440137|T440137]]
* 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s)
* 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
* 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
* 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
* 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1350687{{!}}Rename project namespace on Madurese Wiktionary (T440137)]]
* 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s)
* 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
* 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
* 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
* 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
* 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, [[phab:T416563|T416563]]]
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
* 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
* 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
* 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
* 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
* 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144{{!}}Configure cookie-based enrollment in PHP 8.5 (T441689)]]
* 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
* 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
* 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
* 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for [[gerrit:1362080{{!}}Re-enable NewUserMessage for viwikivoyage (T442485)]]
* 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
* 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
* 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
* 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
* 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
* 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
* 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
* 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
* 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
* 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
* 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
* 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
* 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
* 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
* 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
* 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
* 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
* 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
* 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
* 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
* 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
* 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) (duration: 11m 06s)
* 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
* 13:17 moritzm: installing expat security updates
* 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
* 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
* 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
* 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
* 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see ([[phab:T439885|T439885]]) ([[gerrit:1363090{{!}}1.48.0-wmf.1]], [[gerrit:1363091{{!}}1.48.0-wmf.3]])
* 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
* 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
* 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
* 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
* 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
* 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
* 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
* 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004&quot;
* 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
* 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
* 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
* 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
* 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
* 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
* 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
* 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
* 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
* 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
* 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
* 12:45 claime: donne with poolcounter reboots
* 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
* 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s)
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
* 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
* 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
* 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
* 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:37 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
* 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
* 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
* 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
* 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363106{{!}}ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]]
* 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
* 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
* 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
* 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
* 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
* 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
* 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s)
* 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
* 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
* 12:20 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363105{{!}}ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]]
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
* 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
* 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
* 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
* 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
* 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
* 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
* 12:13 moritzm: installing apt security updates
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
* 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
* 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
* 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
* 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s)
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
* 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
* 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
* 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
* 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
* 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
* 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
* 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
* 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
* 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
* 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
* 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
* 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363104{{!}}ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]]
* 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
* 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
* 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
* 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
* 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s)
* 11:44 tappof: putting metamonitoring checks into &quot;maintenance mode without sending a notification&quot; to update the metamonitoring script [[phab:T443406|T443406]]
* 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
* 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
* 11:41 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363102{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103{{!}}ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]]
* 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
* 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
* 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
* 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
* 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s)
* 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
* 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
* 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
* 11:18 cgoubert@deploy2003: cgoubert: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for [[gerrit:1363062{{!}}ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]]
* 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
* 11:14 claime: Starting poolcounter reboot sequence
* 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s)
* 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:04 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
* 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1363009{{!}}Better browser support for DASH (T442988)]] (1.48.0-wmf.3)
* 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
* 10:49 moritzm: installing vim security updates
* 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
* 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
* 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
* 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
* 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
* 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
* 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
* 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
* 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
* 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
* 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
* 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
* 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
* 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
* 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
* 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
* 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
* 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: [[phab:T440177|T440177]] offline reindex test
* 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
* 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
* 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
* 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
* 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
* 08:15 Tran: Deployed patch for [[phab:T441775|T441775]]
* 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
* 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
* 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
* 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
* 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
* 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
* 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
* 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
* 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
* 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
* 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
* 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
* 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
* 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
* 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
* 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
* 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
* 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
* 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
* 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
* 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
* 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
* 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
* 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
* 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
* 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
* 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
* 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
* 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
* 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
* 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
* 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
* 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
* 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
* 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
* 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
* 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
* 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
* 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
* 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
* 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
* 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
* 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
* 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
* 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
* 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
* 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
* 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
* 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
* 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
* 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
* 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
* 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
* 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
* 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
* 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
* 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
* 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
* 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
* 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
* 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
* 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
* 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
* 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
* 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
* 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
* 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
* 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
* 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
* 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
* 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s)
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
* 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
* 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
* 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
* 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
* 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
* 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
* 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
* 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
* 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
* 00:05 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1362212{{!}}Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217{{!}}Enable chunked videos on testwiki (T373546)]]
* 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync

== 2026-11-03 ==
* 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
* 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
* 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
* 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
* 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
* 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
* 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
* 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
* 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
* 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
* 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
* 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
* 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
* 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
* 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
* 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] (duration: 14m 56s)
* 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
* 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
* 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
* 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
* 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
* 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
* 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
* 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
* 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
* 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
* 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
* 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
* 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
* 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
* 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
* 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
* 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for [[gerrit:1352303{{!}}CommonSettings: remove redundant CodeMirror config settings (T440077)]], [[gerrit:1358261{{!}}metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)]]
* 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
* 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
* 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
* 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
* 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
* 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
* 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] (duration: 10m 49s)
* 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
* 22:38 kemayo@deploy2003: kemayo: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:33 kemayo@deploy2003: Started scap sync-world: Backport for [[gerrit:1362215{{!}}VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)]]
* 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) (duration: 13m 25s)
* 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 22:18 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
* 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1362101{{!}}styles: Restore style to Special:MobileOptions (T363032)]] (1.48.0-wmf.3), [[gerrit:1361724{{!}}Revert &quot;footer: Hide separator before a hidden last footer place&quot; (T387797)]] (1.48.0-wmf.3), [[gerrit:1362134{{!}}downloadPageAction: Add the download link before the bookmark (T443387)]] (1.48.0-wmf.3)
* {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_lik}}
* 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
* 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
* 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
* 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
* 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
* {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Reve}}
* {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for [[gerrit:1361843{{!}}eswiki: add editprotected to botadmin (T443272)]], [[gerrit:1361837{{!}}liwiktionary: Create Appendix namespace (T443269)]], [[gerrit:1361447{{!}}Stop exempting item/property creation from hCaptcha (T440832)]], [[gerrit:1355485{{!}}Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054)]], [[gerrit:1362185{{!}}Revert &quot;Reapply &quot;cirrus: Send more_like}}
* 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
* 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] (duration: 11m 41s)
* 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
* 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
* 21:09 sbassett@deploy2003: sbassett: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
* 21:05 sbassett@deploy2003: Started scap sync-world: Backport for [[gerrit:1360542{{!}}Enable DBSC report-only mode on group0 and test2wiki (T393212)]]
* 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
* 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
* 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
* 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
* 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
* 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
* 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
* 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
* 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
* 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
* 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
* 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
* 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
* 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
* 19:29 dancy@deploy2003: Installation of scap version &quot;4.313.0&quot; completed for 2 hosts
* 19:27 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
* 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 19:17 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:16 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 1 host(s)
* 19:14 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
* 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
* 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
* 19:11 dancy@deploy2003: Installing scap version &quot;4.313.0&quot; for 2 host(s)
* 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
* 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
* 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
* 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
* 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
* 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
* 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
* 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
* 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
* 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
* 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
* 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
* 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
* 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
* 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
* 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
* 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] (duration: 10m 40s)
* 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
* 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
* 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
* 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:16 akhatun@deploy2003: akhatun: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
* 18:11 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1362126{{!}}revision feature counts: increase timeout 20s to 30s (T443482)]]
* 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] (duration: 19m 00s)
* 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
* 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
* 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
* 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
* 17:50 tchanders@deploy2003: tchanders: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:46 tchanders@deploy2003: Started scap sync-world: Backport for [[gerrit:1362074{{!}}Enable link spam queue on Special:AbuseReview for alpha testing (T443432)]]
* 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
* 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
* 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
* 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]] (duration: 14m 43s)
* 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
* 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:28 dancy@deploy2003: Installation of scap version &quot;4.312.0&quot; completed for 2 hosts
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:26 dancy@deploy2003: Installing scap version &quot;4.312.0&quot; for 2 host(s)
* 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
* 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for [[phab:T440968|T440968]]
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P&lt;nowiki&gt;{&lt;/nowiki&gt;ml-serve101[2-5]*&lt;nowiki&gt;}&lt;/nowiki&gt; and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
* 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
* 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
* 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
* 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
* 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
* 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
* 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
* 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
* 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]] (duration: 11m 24s)
* 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
* 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
* 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for [[phab:T440968|T440968]]
* 15:40 dancy@deploy2003: Installation of scap version &quot;4.311.0&quot; completed for 155 hosts
* 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
* 15:36 dancy@deploy2003: Installing scap version &quot;4.311.0&quot; for 155 host(s)
* 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
* 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] (duration: 33m 48s)
* 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
* 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
* 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
* 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
* 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
* 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
* 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - [[phab:T439676|T439676]]
* 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
* 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
* 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
* 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master ([[gerrit:1361885{{!}}1.48.0-wmf.1]], [[gerrit:1361888{{!}}1.48.0-wmf.3]]), [[gerrit:1362064{{!}}Revert &quot;Enable NewUserMessage for viwikivoyage&quot;]]
* 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
* 14:59 samtar@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355072{{!}}Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179)]], [[gerrit:1361887{{!}}Add missing groupOverrides for sysop and bot on ltwiki. (T440216)]] (duration: 21m 43s)
* 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
* 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
* 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
* 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
* 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
* 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
* 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
* 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
* 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
* 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
* 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3  refs [[phab:T442221|T442221]]
* 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
* 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
* 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
* 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
* 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
* 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
* 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
* 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
* 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
* 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
* 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
* 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
* 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
* 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
* 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
* 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
* 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
* 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
* 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
* 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
* 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
* 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) (duration: 10m 58s)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004&quot;
* 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
* 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
* 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 11:50 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
* 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
* 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
* 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361850{{!}}Add --listfile to transcode maint scripts (T435744 T373546)]] (1.48.0-wmf.1)
* 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
* 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
* 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
* 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
* 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
* 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
* 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
* 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
* 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
* 11:12 moritzm: installing chromium security updates
* 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
* 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
* 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
* 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
* 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
* 10:53 jmm@dns1004: END - running authdns-update
* 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
* 10:51 jmm@dns1004: START - running authdns-update
* 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
* 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
* 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
* 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
* 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
* 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
* 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97106)
* 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
* 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
* 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
* 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
* 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]]
* 10:22 moritzm: installing git-lfs security updates
* 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose  # [[phab:T438366|T438366]] (dblist: https://phabricator.wikimedia.org/P97103)
* 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
* 10:15 moritzm: installing imagemagick security updates
* 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
* 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
* 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
* 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
* 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
* 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
* 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
* 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
* 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
* 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
* 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
* 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
* 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
* 09:02 Tran: Deployed patch for [[phab:T441675|T441675]]
* 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
* 08:24 denisse: increasing disk size in arclamp1001
* 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
* 08:13 denisse: incresing disk in arclamp2001
* 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
* 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] (duration: 24m 34s)
* 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
* 04:24 krinkle@deploy2003: krinkle: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:16 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1311061{{!}}MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)]]
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
* 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
* 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
* 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
* 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
* 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
* 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
* 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
* 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
* 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
* 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
* 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
* 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
* 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
* 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
* 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
* 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
* 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
* 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
* 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
* 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
* 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
* 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
* 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
* 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
* 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
* 00:10 maryum: Deployed security fix for [[phab:T440469|T440469]], [[phab:T441384|T441384]], [[phab:T440735|T440735]], [[phab:T440501|T440501]], [[phab:T439716|T439716]]

== 2026-11-02 ==
* 23:24 maryum: Deployed security fix for [[phab:T440502|T440502]], [[phab:T440330|T440330]], [[phab:T442830|T442830]], [[phab:T442861|T442861]]
* 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s)
* 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
* 22:15 krinkle@deploy2003: krinkle: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 22:11 krinkle@deploy2003: Started scap sync-world: Backport for [[gerrit:1361554{{!}}Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555{{!}}Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556{{!}}Draw no outer line for a nested \hline]] (1.48.0-wmf.1)
* 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s)
* 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:56 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1359878{{!}}Enable Vector CTA on all wikipedias (T441705)]]
* 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s)
* 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
* 21:36 jdlrobson@deploy2003: jdlrobson: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
* 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for [[gerrit:1360497{{!}}Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498{{!}}Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551{{!}}Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1)
* 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
* 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
* 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
* 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
* 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
* 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
* 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
* 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
* 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
* 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
* 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
* 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
* 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
* 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
* 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
* 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
* 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
* 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
* 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
* 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
* 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
* 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
* 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
* 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
* 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
* 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
* 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
* 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
* 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
* 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
* 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
* 18:57 dancy@deploy2003: Installation of scap version &quot;4.310.0&quot; completed for 2 hosts
* 18:55 dancy@deploy2003: Installing scap version &quot;4.310.0&quot; for 2 host(s)
* 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s)
* 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
* 18:47 akhatun@deploy2003: akhatun: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:43 akhatun@deploy2003: Started scap sync-world: Backport for [[gerrit:1361525{{!}}Prefetch the LAC revision_feature_counts artifact on all wikipedias]]
* 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
* 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
* 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
* 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
* 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
* 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
* 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
* 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
* 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]] (duration: 07m 48s)
* 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 [[phab:T436338|T436338]]
* 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s)
* 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
* 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
* 17:51 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
* 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
* 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
* 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361521{{!}}Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519{{!}}rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1)
* 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
* 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
* 17:45 dancy: Investigating spiderpig
* 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
* 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
* 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
* 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - [[phab:T440968|T440968]]
* 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
* 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
* 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s)
* 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
* 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
* 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
* 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
* 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
* 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 17:03 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361508{{!}}Temporarily increase circuit breaking (T443274)]]
* 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
* 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
* 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
* 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
* 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
* 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
* 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
* 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
* 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
* 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
* 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
* 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
* 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
* 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
* 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
* 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
* 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
* 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
* 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
* 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
* 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
* 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
* 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
* 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
* 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
* 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
* 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
* 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
* 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
* 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
* 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
* 15:13 Lucas_WMDE: UTC afternoon backport+config window done
* 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s)
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
* 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
* 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
* 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
* 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
* 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361148{{!}}eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406{{!}}GrowthExperiments: enable SpecialHome on testwiki (T442733)]]
* 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
* {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie}}
* 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
* 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
* 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
* 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
* 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
* 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
* {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license ackno}}
* 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
* 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
* 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
* {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for [[gerrit:1361372{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373{{!}}Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379{{!}}Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378{{!}}Set license acknowledgement cookie f}}
* 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
* 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
* 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
* 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
* 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
* 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
* 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
* 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
* 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
* 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
* 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
* 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
* 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
* 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
* 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
* 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
* 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
* 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
* 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
* 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
* 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
* 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
* 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
* 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
* 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
* 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
* 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
* 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
* 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
* 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
* 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
* 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
* 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
* 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
* 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
* 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
* 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
* 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
* 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
* 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
* 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
* 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
* 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
* 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
* 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
* 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
* 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 ([[phab:T441633|T441633]])', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
* 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
* 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
* 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
* 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
* 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
* 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
* 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
* 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
* 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
* 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
* 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
* 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
* 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
* 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
* 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
* 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
* 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
* 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
* 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
* 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
* 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
* 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
* 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
* 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
* 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
* 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
* 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
* 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
* 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
* 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]]) (duration: 07m 48s)
* 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
* 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version ([[phab:T434925|T434925]])
* 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
* 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
* 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
* 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
* 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
* 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s)
* 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
* 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
* 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
* 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
* 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): [[phab:T437278|T437278]]
* 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1358412{{!}}Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1)
* 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
* 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
* 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
* 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
* 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
* 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
* 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
* 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s)
* 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
* 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
* 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
* 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
* 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
* 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
* 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
* 10:44 ladsgroup@deploy2003: ladsgroup: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for [[gerrit:1361350{{!}}Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1)
* 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
* 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
* 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s)
* 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
* 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
* 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
* 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
* 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
* 09:10 brouberol@dns1004: END - running authdns-update
* 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
* 09:08 brouberol@dns1004: START - running authdns-update
* 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for [[gerrit:1355410{{!}}Turn on Wikibase hCaptcha flag for Beta (T440832)]]
* 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] (duration: 36m 41s)
* 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
* 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for [[gerrit:1359782{{!}}Enable sub-references globally (T440304)]]
* 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
* 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
* 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync

== 2026-11-01 ==
* 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
* 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
* 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
* 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
* 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
* 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
* 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
* 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>2coqls9geqbxf6p1mfbr17bazx0m48p</sha1>
    </revision>
  </page>
  <page>
    <title>Help talk:Toolforge/Toolforge standards committee</title>
    <ns>13</ns>
    <id>311299</id>
    <revision>
      <id>2479004</id>
      <parentid>2475995</parentid>
      <timestamp>2026-11-07T14:03:25Z</timestamp>
      <contributor>
        <username>Lucas Werkmeister</username>
        <id>8093</id>
      </contributor>
      <comment>/* Nominations */ nominate [[User:JJMC89]]</comment>
      <origin>2479004</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="13534" sha1="4uzmhmgzt5i9hxclqvp20vhdo182gnm" xml:space="preserve">== December 2016 committee nominations ==
{{discussion top}}
The following Tool Labs members have been nominated for appointment to the Tool Labs standards committee. Self-nominations are acceptable. Nominations by a third party should be counter-signed by the nomination to indicate they would accept the responsibility and be willing to sign the [[Volunteer NDA|Volunteer NDA]].

{| class=&quot;wikitable&quot;
! scope=&quot;col&quot; | Nominee
! scope=&quot;col&quot; | Nominated by
! scope=&quot;col&quot; | Counter signed if not self nominated
|-
| {{u|Harej}}
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
| [[User:Harej|Harej]] ([[User talk:Harej|talk]]) 08:13, 15 December 2016 (UTC)
|-
| {{u|Multichill}}
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
|
|-
| {{u|Matanya}}
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
|[[User:Matanya|matanya]] ([[User talk:Matanya|talk]]) 12:59, 15 December 2016 (UTC)
|-
| {{u|Cyberpower678}}
| Self-nom
|
|-
| {{u|Zhuyifei1999}} 
| Self-nom
|
|-
| {{u|Magnus Manske}}
| [[User:Yuvipanda|yuvipanda]] ([[User talk:Yuvipanda|talk]])
|
|-
| {{u|Quiddity}}
| [[User:Yuvipanda|yuvipanda]] ([[User talk:Yuvipanda|talk]])
| [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]]) 19:20, 21 December 2016 (UTC)
|-
| {{u|Tom29739}}
| [[User:Yuvipanda|yuvipanda]] ([[User talk:Yuvipanda|talk]])
| [[User:Tom29739|Tom29739]] ([[User talk:Tom29739|talk]]) 00:17, 5 January 2017 (UTC)
|-
| {{u|Jean-Frédéric}}
| [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]])
| [[User:Jean-Frédéric|Jean-Frédéric]] ([[User talk:Jean-Frédéric|talk]]) 13:14, 8 January 2017 (UTC)
|-
| {{u|Lokal Profil}}
| [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]])
|
|-
| {{u|Huji}}
| [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]])
| [[User:Huji|huji]] ([[User talk:Huji|talk]]) 00:23, 25 December 2016 (UTC)
|-
| {{u|Eranroz}}
| [[User:Quiddity|Quiddity]] ([[User talk:Quiddity|talk]])
| [[User:Eranroz|eranroz]] ([[User talk:Eranroz|talk]]) 23:43, 24 December 2016 (UTC)
|-
| {{u|Ladsgroup}}
| [[User:Huji|huji]] ([[User talk:Huji|talk]])
| [[User:Ladsgroup|Ladsgroup]] ([[User talk:Ladsgroup|talk]]) 01:20, 27 December 2016 (UTC)
|}
{{discussion bottom}}

== Proposal feedback ==
{{discussion top}}

*I {{support}} this proposal. --[[User:Alex Monk|&lt;span style=&quot;color: orange; font-weight: bold; font-family: Ubuntu;&quot;&gt;Alex Monk&lt;/span&gt;]] &lt;sup&gt;&lt;span style=&quot;font-family: Ubuntu;&quot;&gt;([[User talk:Alex Monk|talk]] &amp;bull; [[Special:Contributions/Alex Monk|contribs]])&lt;/span&gt;&lt;/sup&gt; 22:35, 15 December 2016 (UTC)
*{{support}} [[User:Legoktm|legoktm]] ([[User talk:Legoktm|talk]]) 18:12, 21 December 2016 (UTC)
* {{support}} [[User:Dalba|Dalba]] ([[User talk:Dalba|talk]]) 10:19, 22 December 2016 (UTC)
* {{Support}}.  --[[User:Tim Landscheidt|Tim Landscheidt]] ([[User talk:Tim Landscheidt|talk]]) 22:15, 22 December 2016 (UTC)
* {{support}} [[User:Huji|huji]] ([[User talk:Huji|talk]]) 00:26, 25 December 2016 (UTC)
{{discussion bottom}}

== Good luck and thanks. ==

Nice! — [[user:billinghurst|billinghurst]] ''&lt;span style=&quot;font-size:smaller&quot;&gt;[[user talk:billinghurst|sDrewth]]&lt;/span&gt;'' 11:04, 24 January 2017 (UTC)

== Help with Toolforge tool with unknown license ==

Hi Standards Committee,

I'm in the process of taking over VICbot, a Commons bot created by {{u|Dschwen}} ([https://toolsadmin.wikimedia.org/tools/id/vic toolforge entry]), I'm taking it over because the maintainer isn't very active these days and the bot was not doing its job. The source code is available on-wiki at [[c:User:VICbot/source]], so getting the source code isn't the problem - the problem is that it has no license. Obviously, by publishing on-wiki it's available as CC-BY-SA 3.0, but a) that isn't a great choice for code, and b) that isn't an acceptable license for toolforge. I have contacted the bot's creator both on their talk page about me operating a fork of the bot (last week) and via email asking about licensing (earlier today), but I don't think I'm going to get a response, so I'm not sure how to approach this problem. If someone could see if there is a license in the copy of the tool running on toolforge, that would be very helpful; failing that, any advice on how to deal with this would be much appreciated. [[User:GeneralNotability|GeneralNotability]] ([[User talk:GeneralNotability|talk]]) 01:59, 14 October 2020 (UTC)
: The code is at /data/project/vic/vic_bot on Toolforge, but it does not have a license. —&amp;thinsp;[[User:JJMC89|JJMC89]]&amp;thinsp;&lt;small&gt;([[User talk:JJMC89|T]]'''·'''[[Special:Contributions/JJMC89|C]])&lt;/small&gt; 02:22, 14 October 2020 (UTC)
::{{u|JJMC89}} thanks - didn't realize that would be publicly accessible. Oh well, it was worth a shot. [[User:GeneralNotability|GeneralNotability]] ([[User talk:GeneralNotability|talk]]) 02:52, 14 October 2020 (UTC)

== August 2024 committee nominations ==
{{Tracked|T370474|In progress}}
The Toolforge standards committee (née Tool Labs standards committee) was established in 2017 following a membership [[#December_2016_committee_nominations|nomination process in December 2016]]. There have been no formal membership changes in the subsequent 7 years. Per the original charter of the committee, the process for membership changes is to be handled on an as-needed basis at the discretion of the Wikimedia Foundation.

As a representative of the Wikimedia Foundation, {{U|BryanDavis}} has initiated this nomination process in order to establish a new quorum of three or more active committee members. Any Toolforge project member with a Developer account not blocked in any Wikimedia technical space is eligible for nomination. Self-nominations are acceptable. Nominations by a third party should be counter-signed by the nominee to indicate they would accept the responsibility and be willing to sign the [[Volunteer NDA]]. Nominations will remain open until at least 2024-08-26.

Wikimedia Foundation staff will work with folks who are successfully nominated to pursue the required Volunteer NDA following the close of the nomination period. No community voting phase will be used in this iteration of the process. Appointment to the committee will be at the discretion of the Wikimedia Foundation. Appointees will be made from the list of nominees who accept their nomination and succeed in signing the Volunteer NDA.

{{discussion top|1=
Nominations closed August 26, 2024, End of Day, [[w:Anywhere on Earth|Anywhere on Earth]] (AoE) (2024-08-27T12:00Z). Nominees submitted prior to the close of nominations will be contacted to pursue the required Volunteer NDA before being appointed to the committee. Once established the new committee membership will be announced on [[mail:cloud-announce|cloud-announce@lists.wikimedia.org]].
}}
The following Toolforge members have been nominated for appointment to the Toolforge standards committee:
{| class=&quot;wikitable&quot;
! scope=&quot;col&quot; | Nominee
! scope=&quot;col&quot; | Nominated by
! scope=&quot;col&quot; | Counter signed if not self nominated
|-
| {{u|JJMC89}}
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
| [[User:JJMC89|JJMC89]]&amp;thinsp;&lt;small&gt;([[User talk:JJMC89|T]]'''·'''[[Special:Contributions/JJMC89|C]])&lt;/small&gt; 20:40, 5 August 2024 (UTC)
|-
| {{u|Pintoch}}
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
| [[User:Pintoch|Pintoch]] ([[User talk:Pintoch|talk]]) 14:14, 6 August 2024 (UTC)
|-
| {{u|Lucas Werkmeister}}
| [[User:Lucas Werkmeister|Lucas Werkmeister]] ([[User talk:Lucas Werkmeister|talk]]) 20:33, 10 August 2024 (UTC)
|-
| {{u|Legoktm}}
| [[User:Legoktm|Legoktm]] ([[User talk:Legoktm|talk]]) 05:35, 13 August 2024 (UTC)
|-
| {{u|Waldyrious}}
| [[User:Gergő Tisza|tgr]] ([[User talk:Gergő Tisza|talk]])
| [[User:Waldyrious|Waldyrious]] ([[User talk:Waldyrious|talk]]) 17:02, 13 August 2024 (UTC)
|-
| {{u|SD0001}}
| [[User:SD0001|SD0001]] ([[User talk:SD0001|talk]]) 08:27, 14 August 2024 (UTC)
|-
| {{u|MusikAnimal}}
| [[User:MusikAnimal|MusikAnimal]] ([[User talk:MusikAnimal|talk]]) 20:47, 21 August 2024 (UTC)
|-
| {{u|Theprotonade}}
| [[User:Theprotonade|Theprotonade]] ([[User talk:Theprotonade|talk]]) 23:32, 27 August 2024 (UTC)
|}
{{discussion bottom}}
-- [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]]) 20:23, 5 August 2024 (UTC)

== November 2025 committee nominations ==
{{discussion top|1=Once the required Volunteer NDAs are completed, the new committee will be appointed and announced on cloud-announce@. —&amp;thinsp;[[User:JJMC89|JJMC89]]&amp;thinsp;&lt;small&gt;([[User talk:JJMC89|T]]'''·'''[[Special:Contributions/JJMC89|C]])&lt;/small&gt; 14:01, 23 December 2025 (UTC)}}
Following the [[Special:Diff/2357725|TFSC's revision to the appointment process]] from appointment as needed by the Foundation to an annual appointment by the existing committee, this is the first annual appointment nomination period.

Any Toolforge project member in good standing (not blocked in any Wikimedia technical space) is eligible for nomination. Self-nominations are acceptable. Nominations by a third party must be counter-signed by the nominee to indicate they would accept the responsibility and be willing to sign the [[Volunteer NDA]].

Following the nomination period there will be a two-week period for the Toolforge admins and community to provide feedback on the nominees to the current committee. Following the feedback, the current committee will vote on the nominees. Nominees approved by the committee must sign the Volunteer NDA to be appointed to the committee.

—&amp;thinsp;[[User:JJMC89|JJMC89]]&amp;thinsp;&lt;small&gt;([[User talk:JJMC89|T]]'''·'''[[Special:Contributions/JJMC89|C]])&lt;/small&gt; 18:21, 5 November 2025 (UTC)

=== Nominations ===
The nomination period is open through 2025-11-28.
{| class=&quot;wikitable&quot;
! scope=&quot;col&quot; | Nominee
! scope=&quot;col&quot; | Nominated by
! scope=&quot;col&quot; | Counter signed if not self nominated
|-
| [[User:Sohom Datta|Sohom Datta]]
| Sohom Datta
| 
|-
| [[User:JJMC89|JJMC89]]
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
| [[User:JJMC89|JJMC89]]&amp;thinsp;&lt;small&gt;([[User talk:JJMC89|T]]'''·'''[[Special:Contributions/JJMC89|C]])&lt;/small&gt;
|-
| [[User:Lucas Werkmeister|Lucas Werkmeister]]
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
| [[User:Lucas Werkmeister|Lucas Werkmeister]] ([[User talk:Lucas Werkmeister|talk]])
|-
| [[User:Pintoch|Pintoch]]
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
| [[User:Pintoch|Pintoch]] ([[User talk:Pintoch|talk]])
|-
| [[User:SD0001|SD0001]]
| [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]])
| [[User:SD0001|SD0001]] ([[User talk:SD0001|talk]])
|-
| [[User:EggRoll97|EggRoll97]]
| EggRoll97
| 
|}

=== Nominee feedback ===
Feedback on the nominees by email was open through 2025-12-12. No feedback was received.
{{discussion bottom}}

== Activity ==

Has there been activity of the current committee that is not visible with [https://phabricator.wikimedia.org/feed/query/xniva5gYFqmZ/#R this Phabricator query]? [[User:taavi|taavi]] ([[User talk:Taavi|talk!]]) 18:32, 4 December 2025 (UTC)

:I believe all of the direct [[Help:Toolforge/Abandoned tool policy|abandoned tool policy]] assistance rendered by the committee should be included in that query. The committee's [[mail:toolforge-standards-committee|private mailing list]] discussions and the partial committee meetup during [[mw:Wikimedia Hackathon 2025|Wikimedia Hackathon 2025]] are not captured in Phabricator. The most impactful outcome to date of non-public discussions was the [[Special:Diff/2357725|updated appointment process]] and the subsequent [[#November 2025 committee nominations]] iteration. -- [[User:BryanDavis|BryanDavis]] ([[User talk:BryanDavis|talk]]) 18:50, 4 December 2025 (UTC)

== November 2026 committee nominations ==

This is the annual appointment nomination period for the next iteration of the TFSC, following the [[Help:Toolforge/Toolforge standards committee#Membership changes|TFSC's appointment process]].

Any Toolforge project member in good standing (not blocked in any Wikimedia technical space) is eligible for nomination. Self-nominations are acceptable. Nominations by a third party must be counter-signed by the nominee to indicate they would accept the responsibility and be willing to sign the [[Volunteer NDA]].

Following the nomination period there will be a two-week period for the Toolforge admins and community to provide feedback on the nominees to the current committee. Following the feedback, the current committee will vote on the nominees. Nominees approved by the committee must sign the Volunteer NDA to be appointed to the committee.

—&amp;thinsp;[[User:JJMC89|JJMC89]]&amp;thinsp;&lt;small&gt;([[User talk:JJMC89|T]]'''·'''[[Special:Contributions/JJMC89|C]])&lt;/small&gt; 06:30, 1 November 2026 (UTC)

=== Nominations ===
The nomination period is open through 2026-11-22.
{| class=&quot;wikitable&quot;
! scope=&quot;col&quot; | Nominee
! scope=&quot;col&quot; | Nominated by
! scope=&quot;col&quot; | Counter signed if not self nominated
|-
| {{u|Max}}
| [[User:SD0001|SD0001]] ([[User talk:SD0001|talk]]) 09:53, 1 November 2026 (UTC)
| [[User:Max|Max]] ([[User talk:Max|talk]]) 16:31, 1 November 2026 (UTC)
|-
|{{u|Dolphyb}}
|Dolphyb 
|
|-
| {{u|Lucas Werkmeister}}
| [[User:Lucas Werkmeister|Lucas Werkmeister]] ([[User talk:Lucas Werkmeister|talk]]) 11:08, 1 November 2026 (UTC)
|
|-
| {{u|JJMC89}}
| [[User:Lucas Werkmeister|Lucas Werkmeister]] ([[User talk:Lucas Werkmeister|talk]]) 14:03, 7 November 2026 (UTC)
|
|}

=== Nominee feedback ===
{{note|Feedback is closed until after nominations are closed.}}</text>
      <sha1>4uzmhmgzt5i9hxclqvp20vhdo182gnm</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng/SAL</title>
    <ns>498</ns>
    <id>443628</id>
    <revision>
      <id>2478975</id>
      <parentid>2478958</parentid>
      <timestamp>2026-11-07T12:31:53Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819762 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478975</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="54133" sha1="g0iraaqf4wcmfej8pqdpm9wp9dsqc2s" xml:space="preserve">=== 2026-11-07 ===
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819762 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251009 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 16:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695823 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249636648 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248508521 (https://github.com/cluebotng/component-configs/commits/3acae96966e16f4f9f91683ad36f5e56d329753a)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234031825 (https://github.com/cluebotng/component-configs/commits/15c352e208f001eee46b8462013282f7e93d9872)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233327546 (https://github.com/cluebotng/component-configs/commits/e9471c88503e10f4ba84992f33bfa015a2c6c887)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248920 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188059 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 13:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42222101673 (https://github.com/cluebotng/component-configs/commits/5a9290a457af2e5700d955bc65fbd8d77cad712c)
* 12:21 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485438 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 10:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42200307462 (https://github.com/cluebotng/component-configs/commits/255eec1d87ca571dcfd0f09253a9504e7d2b70f1)
* 07:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181945119 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270625 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297609 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42166677976 (https://github.com/cluebotng/component-configs/commits/ceeb81f4d5e5d7b6a4947b95df020093bdea2903)
* 04:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42162423731 (https://github.com/cluebotng/component-configs/commits/f0d701fcec7061cd6d2bf7ea127a73895e4a005f)
* 03:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42155618053 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 05:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41976470110 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974543482 (https://github.com/cluebotng/component-configs/commits/7f5f130ad27cc6b58d18dc7db9fb6f17a994795f)
* 05:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300376 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 04:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41970293977 (https://github.com/cluebotng/component-configs/commits/249d7f9d3e64c31431d63973b2e535b32af4e788)
* 03:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41961409161 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 23:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942732 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214126 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107372 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a)
* 04:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41773053080 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208829 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274447 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064641 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2)
* 00:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119511 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580770 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7)
* 12:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616293 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215687 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b)
* 05:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264002 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147096 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696228 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e)
* 02:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216287 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434850 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-11-01 ===
* 21:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41350442746 (https://github.com/cluebotng/component-configs/commits/9f8d02aa99dd4b425755d1cf842fbce1fa045266)
* 20:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41346673895 (https://github.com/cluebotng/component-configs/commits/eea2ac3e027177d39bf8015524cc6b02f32445f9)
* 12:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41306290063 (https://github.com/cluebotng/component-configs/commits/d2705aba98618c9fa1aecc8578e6861ed2ef9ef8)

=== 2026-10-31 ===
* 16:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41219859885 (https://github.com/cluebotng/component-configs/commits/ade6a3988f2a8e1609e94a58b7cb546484dd8e9a)
* 14:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074802 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347107 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603041 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191380192 (https://github.com/cluebotng/component-configs/commits/2be620aa8d34f581c9c9bb72401e2bcb82705c7d)
* 09:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726845 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 09:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186724198 (https://github.com/cluebotng/component-configs/commits/8474642453be7b488acccbcd8e7bfe5261236e6c)
* 08:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002299 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139892801 (https://github.com/cluebotng/component-configs/commits/ce1c920d2a60015846167ad5fba607bfa99e27fa)
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595319 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610828 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626173 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41135430856 (https://github.com/cluebotng/component-configs/commits/118e964d982be74b124949a21f67c966a2b2bea0)
* 21:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41131252248 (https://github.com/cluebotng/component-configs/commits/9aadd326a870be6402adf95e5254a733fdc7c0ac)
* 21:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558288 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761927 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)

=== 2026-10-29 ===
* 15:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40903315370 (https://github.com/cluebotng/component-configs/commits/48d12f535986ffcf04bdc857a204667a10e4579d)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696699 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902240936 (https://github.com/cluebotng/component-configs/commits/77ffbc8ebcebe0ba0b96808c0324623bd61aadf9)
* 15:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900617624 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898273109 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898254138 (https://github.com/cluebotng/component-configs/commits/5392cc3d04cf9e3086473ded50ae76eb8ffb6a29)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40894576159 (https://github.com/cluebotng/component-configs/commits/d180ba6310a8569d7c091af3c9dd92dcdbda1991)

=== 2026-10-28 ===
* 18:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217792 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe)
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507911 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788312 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987512 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879022 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)
* 02:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021123 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf)

=== 2026-10-27 ===
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112168 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846790 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 15:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588912 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106)
* 14:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783783 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 14:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784252 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108700 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591155 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231026 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21)
* 09:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261201 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249547 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601906 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed)
* 06:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40459540107 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798083 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 05:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40455721193 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 05:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518044 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826128 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9)
* 04:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765772 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375412 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527027 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444184085 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)
* 03:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895975 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736567 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6)

=== 2026-10-26 ===
* 23:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679243 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a)
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40378908239 (https://github.com/cluebotng/component-configs/commits/2d1531d1cbfbfa07a816f43b825ad9d8214f9996)

=== 2026-10-23 ===
* 22:56 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-09-03 ===
* 10:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33742087870 (https://github.com/cluebotng/component-configs/commits/781284a9a6b62d000c7c570fb34d20bbe21933a0)

=== 2026-09-01 ===
* 19:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856122 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3)
* 16:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531639211 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-25 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32866638493 (https://github.com/cluebotng/component-configs/commits/f0c0373d1612550e858604e6c40555d78d071318)
* 14:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32859995726 (https://github.com/cluebotng/component-configs/commits/8e54667ba485459bad6bd7798274c98ea487caee)

=== 2026-08-24 ===
* 09:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32713575575 (https://github.com/cluebotng/component-configs/commits/15461b10d7f124d327ec6065e11b3fce0465cc68)

=== 2026-07-30 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/30522988957 (https://github.com/cluebotng/component-configs/commits/e38d48bc554d39c0874c954226583b005318f76f)

=== 2026-07-23 ===
* 11:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/30003154188 (https://github.com/cluebotng/component-configs/commits/97520d86ff57924e8237a0446010f837e79b9062)
* 10:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29998195703 (https://github.com/cluebotng/component-configs/commits/e8762796572d00e69633a1b076413d5aa4ee3508)
* 09:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29995256262 (https://github.com/cluebotng/component-configs/commits/c08c3e19100546a5f8ed26a2bfa7095cb4633147)

=== 2026-07-20 ===
* 15:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29753268267 (https://github.com/cluebotng/component-configs/commits/3cd242ed54e9146f6a3dfd9305f97d4580c2fc06)
* 14:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29749326972 (https://github.com/cluebotng/component-configs/commits/83d05d6d10640a88d80bdbc041955ca27603a0fe)
* 13:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29744681881 (https://github.com/cluebotng/component-configs/commits/8b5c46ef715a2e57f383d35d7acc37bfbfa4db2d)
* 11:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29738720465 (https://github.com/cluebotng/component-configs/commits/661b4bae545b221090aa694fa30ff924415a5b22)
* 11:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29737812391 (https://github.com/cluebotng/component-configs/commits/10014b40131fb0e2f27f7de26102f6689287add4)

=== 2026-07-16 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29484727917 (https://github.com/cluebotng/component-configs/commits/328ca21897e26730c95c4dad77c34d5e1bbf9c31)
* 08:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29482938340 (https://github.com/cluebotng/component-configs/commits/6ddd234f9ddace81ab9e1e7e45b5f974c5938a11)

=== 2026-07-15 ===
* 17:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29436632270 (https://github.com/cluebotng/component-configs/commits/ff37e7e6d3e126ba24d186be144aeae3d2048954)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29434331433 (https://github.com/cluebotng/component-configs/commits/37dd33ec5b56f991e912ec44ce96a536735cb1a0)
* 16:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29430449675 (https://github.com/cluebotng/component-configs/commits/ecf4f749293c44e72e74858bef8193eb29695c9a)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29428509974 (https://github.com/cluebotng/component-configs/commits/842854f79b07dc4ec739505a99c8cb11009f8f4f)
* 15:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29427447334 (https://github.com/cluebotng/component-configs/commits/0d2bc5e6c57ba81a40584a85fe4ea3868875999f)
* 08:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29399091032 (https://github.com/cluebotng/component-configs/commits/5880b5b641021f572e9dd6a3d76443db4a607114)

=== 2026-07-14 ===
* 08:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29318761866 (https://github.com/cluebotng/component-configs/commits/04593c4734db7ed505ba34446adacc22176c18ed)

=== 2026-07-03 ===
* 13:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28663718696 (https://github.com/cluebotng/component-configs/commits/8b35b36bf642f83b59af6cf2903f9fcdd6d67007)
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28651139847 (https://github.com/cluebotng/component-configs/commits/11ad538e5876626229416e2ba16bc9f7a7e5cace)
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650658730 (https://github.com/cluebotng/component-configs/commits/0d3a15b63afd3afe5965be4754401712df0b4ae1)
* 09:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650009604 (https://github.com/cluebotng/component-configs/commits/a66ec11b207598051416135000b2a5815ba30080)

=== 2026-07-02 ===
* 22:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28625556671 (https://github.com/cluebotng/component-configs/commits/e95434d48b96490f3526640acde23ce59116589b)
* 21:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412770 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 20:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28618176241 (https://github.com/cluebotng/component-configs/commits/c3b2ebd135508a9a0f781ed332514069fb4d8c89)

=== 2026-07-01 ===
* 20:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28544657461 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861)
* 18:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28538748543 (https://github.com/cluebotng/component-configs/commits/6c7bcbaa2a10eb01c07a4df9fe5c1ea3b571b351)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28536333606 (https://github.com/cluebotng/component-configs/commits/fc7f0d0647c84e0e5c9f02d279015dc3183f3151)
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28525308526 (https://github.com/cluebotng/component-configs/commits/4d35560ba5b135c6dff161ca5a7ed6c393edd180)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506476 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53)

=== 2026-06-30 ===
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53)

=== 2026-06-29 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28404836827 (https://github.com/cluebotng/component-configs/commits/637e933186e3a7fedd0e7328d4c4e6035d12ec82)

=== 2026-06-27 ===
* 22:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337267 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 13:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28291090339 (https://github.com/cluebotng/component-configs/commits/feff80c249bc28b62ed154c74e8f8b1bfa597e3c)
* 13:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28291037948 (https://github.com/cluebotng/component-configs/commits/03f0e1e95886611fe4bf68c103a51f34ec7a8f41)

=== 2026-06-24 ===
* 18:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 17:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 16:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 14:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28103726388 (https://github.com/cluebotng/component-configs/commits/3ae736249e7b8cae3c57261c55b221a4d62296f9)

=== 2026-06-16 ===
* 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27639941201 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849976 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737553 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634022645 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770908 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 14:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179755 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-13 ===
* 13:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463956 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)

=== 2026-06-10 ===
* 15:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125849 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27279260980 (https://github.com/cluebotng/component-configs/commits/83e16c74dca286ed8f7104d49a271dee18c41854)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788838 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276699371 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510139 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411819 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397723 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56)
* 12:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339407 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225356702 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156804 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601458 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)

=== 2026-06-04 ===
* 22:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d)

=== 2026-06-01 ===
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978145 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757972034 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864800 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862186 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719590163 (https://github.com/cluebotng/component-configs/commits/9fc63fa520f0c2c3790d3d3236682dbb83382a9f)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575482 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d)

=== 2026-05-29 ===
* 00:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639432 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20)

=== 2026-05-28 ===
* 18:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792702 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8)

=== 2026-05-21 ===
* 20:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627543 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056)

=== 2026-05-19 ===
* 00:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076424 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c)

=== 2026-05-17 ===
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25986406214 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752565 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616046 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25727821548 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842111 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562306 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480578 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280492 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-07 ===
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25505647057 (https://github.com/cluebotng/component-configs/commits/874d7f6f407fc9a3995f52f40312cd7d3a712176)

=== 2026-05-06 ===
* 18:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754913 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-04 ===
* 07:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-05-02 ===
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278570 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)
* 00:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24865518502 (https://github.com/cluebotng/component-configs/commits/6e953f35fdca38226cde9ea7280f948f34242881)

=== 2026-04-23 ===
* 23:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24863962598 (https://github.com/cluebotng/component-configs/commits/07b9ad8616853af2ed49f96de6e55c14e3faabe0)
* 09:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24827343249 (https://github.com/cluebotng/component-configs/commits/218a7fa56222cf3e98b642eebbf6b1b2b273a92d)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826859245 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79)
* 09:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826656498 (https://github.com/cluebotng/component-configs/commits/90aa39b3e81f19f42d08d5ec6131ba09c68bd786)

=== 2026-04-17 ===
* 00:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24540678933 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24422836823 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442412 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24249897963 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247614873 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c)
* 14:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247570676 (https://github.com/cluebotng/component-configs/commits/93ced49392782bf65e34d13f10cbeaafa760f115)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093216 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656823 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659535101 (https://github.com/cluebotng/component-configs/commits/b825610c7f23870a8561b00b5f8546b107643015)

=== 2026-03-25 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729395 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640340 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551615 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381633193 (https://github.com/cluebotng/component-configs/commits/ac40c461942f4541b640a32ff0141268418abc12)

=== 2026-03-19 ===
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23315406679 (https://github.com/cluebotng/component-configs/commits/fd07020c08545c83ab35667616a26081966648df)

=== 2026-02-15 ===
* 10:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380291 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb)

=== 2025-11-11 ===
* 15:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301650 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266856634 (https://github.com/cluebotng/component-configs/commits/b0e9170597a778654185be762c580e2a6e19492f)

=== 2025-11-06 ===
* 01:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19121350626 (https://github.com/cluebotng/component-configs/commits/80f69f0ab7b09c2e3e5e208d847a954cb1975bc6)

=== 2025-11-05 ===
* 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995211 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9)
* 18:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19112506240 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)
* 16:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108852168 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217917 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872623 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267293 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-08 ===
* 15:47 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.1
* 11:23 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.0

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101448 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 09:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092350259 (https://github.com/cluebotng/component-configs/commits/283965c9240c0c5a72e0ea1203439583935295cb)

=== 2025-09-27 ===
* 13:07 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.3.0

=== 2025-09-26 ===
* 18:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649036 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca)
* 12:52 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.4
* 12:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18038096319 (https://github.com/cluebotng/component-configs/commits/e10e601b4fb06b3fd97856ef86a30e5391fb4f17)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926507 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d)
* 12:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037156332 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)

=== 2025-09-25 ===
* 21:20 wmbot~damian@tools-bastion-15: bot deployed @ v1.2.2
* 21:20 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.2
* 19:17 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.1
* 19:16 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.0
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18017003184 (https://github.com/cluebotng/component-configs/commits/e9fc8d46ac1a0ff0ac6203458fa171c6430492ce)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016894528 (https://github.com/cluebotng/component-configs/commits/cfc9adc9516df0f11c8b6d1df68232d0a46cb4eb)
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998407 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 17:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801881 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221965 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782359 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677281 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531091 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)

=== 2025-09-22 ===
* 19:08 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.3
* 19:04 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.2
* 19:02 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.1

=== 2025-08-29 ===
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.2.0
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.1

=== 2025-08-15 ===
* 21:14 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.0
* 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/1.0.34
* 12:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.33
* 00:23 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.32
* 00:08 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.31

=== 2025-08-14 ===
* 16:13 wmbot~damian@tools-bastion-13: irc-relay deployed @ v1.1.12
* 12:47 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2
* 12:43 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2

=== 2025-08-11 ===
* 13:06 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.3
* 12:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.30

=== 2025-08-10 ===
* 17:42 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.2
* 17:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.29
* 16:35 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.1

=== 2025-08-08 ===
* 15:37 wmbot~damian@tools-bastion-12: Updated ci-execute-fabric to use dedicated unix account, dropped key from human account
* 15:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.28
* 15:33 wmbot~damian@tools-bastion-13: report deployed @ refs/tags/v1.0.28

=== 2025-08-07 ===
* 15:24 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27

=== 2024-03-21 ===
* 10:13 dcaro: fixed .lighttpd.conf file to add port and remove socket

=== 2021-07-23 ===
* 16:40 majavah: stop cbng_relay grid job, still having issues with irc connection - [[phab:T274871|T274871]]

=== 2020-01-13 ===
* 20:53 wm-bot: &lt;root&gt; Restarted webservice to fix broken registration with the front proxy ([[phab:T242538|T242538]])

=== 2019-03-07 ===
* 05:06 bd808: Killed cbng_bot job stuck in deletion state with 4000+ zombie child processes ([[phab:T217817|T217817]])


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>g0iraaqf4wcmfej8pqdpm9wp9dsqc2s</sha1>
    </revision>
    <revision>
      <id>2478989</id>
      <parentid>2478975</parentid>
      <timestamp>2026-11-07T12:37:49Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856570 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)</comment>
      <origin>2478989</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="54339" sha1="kkatwlf2o78hpmh6brfgxiysb1awz5m" xml:space="preserve">=== 2026-11-07 ===
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856570 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819762 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251009 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 16:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695823 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249636648 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248508521 (https://github.com/cluebotng/component-configs/commits/3acae96966e16f4f9f91683ad36f5e56d329753a)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234031825 (https://github.com/cluebotng/component-configs/commits/15c352e208f001eee46b8462013282f7e93d9872)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233327546 (https://github.com/cluebotng/component-configs/commits/e9471c88503e10f4ba84992f33bfa015a2c6c887)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248920 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188059 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 13:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42222101673 (https://github.com/cluebotng/component-configs/commits/5a9290a457af2e5700d955bc65fbd8d77cad712c)
* 12:21 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485438 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 10:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42200307462 (https://github.com/cluebotng/component-configs/commits/255eec1d87ca571dcfd0f09253a9504e7d2b70f1)
* 07:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181945119 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270625 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297609 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42166677976 (https://github.com/cluebotng/component-configs/commits/ceeb81f4d5e5d7b6a4947b95df020093bdea2903)
* 04:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42162423731 (https://github.com/cluebotng/component-configs/commits/f0d701fcec7061cd6d2bf7ea127a73895e4a005f)
* 03:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42155618053 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 05:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41976470110 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974543482 (https://github.com/cluebotng/component-configs/commits/7f5f130ad27cc6b58d18dc7db9fb6f17a994795f)
* 05:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300376 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 04:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41970293977 (https://github.com/cluebotng/component-configs/commits/249d7f9d3e64c31431d63973b2e535b32af4e788)
* 03:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41961409161 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 23:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942732 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214126 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107372 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a)
* 04:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41773053080 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208829 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274447 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064641 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2)
* 00:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119511 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580770 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7)
* 12:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616293 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215687 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b)
* 05:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264002 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147096 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696228 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e)
* 02:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216287 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434850 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-11-01 ===
* 21:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41350442746 (https://github.com/cluebotng/component-configs/commits/9f8d02aa99dd4b425755d1cf842fbce1fa045266)
* 20:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41346673895 (https://github.com/cluebotng/component-configs/commits/eea2ac3e027177d39bf8015524cc6b02f32445f9)
* 12:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41306290063 (https://github.com/cluebotng/component-configs/commits/d2705aba98618c9fa1aecc8578e6861ed2ef9ef8)

=== 2026-10-31 ===
* 16:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41219859885 (https://github.com/cluebotng/component-configs/commits/ade6a3988f2a8e1609e94a58b7cb546484dd8e9a)
* 14:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074802 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347107 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603041 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191380192 (https://github.com/cluebotng/component-configs/commits/2be620aa8d34f581c9c9bb72401e2bcb82705c7d)
* 09:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726845 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 09:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186724198 (https://github.com/cluebotng/component-configs/commits/8474642453be7b488acccbcd8e7bfe5261236e6c)
* 08:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002299 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139892801 (https://github.com/cluebotng/component-configs/commits/ce1c920d2a60015846167ad5fba607bfa99e27fa)
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595319 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610828 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626173 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41135430856 (https://github.com/cluebotng/component-configs/commits/118e964d982be74b124949a21f67c966a2b2bea0)
* 21:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41131252248 (https://github.com/cluebotng/component-configs/commits/9aadd326a870be6402adf95e5254a733fdc7c0ac)
* 21:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558288 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761927 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)

=== 2026-10-29 ===
* 15:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40903315370 (https://github.com/cluebotng/component-configs/commits/48d12f535986ffcf04bdc857a204667a10e4579d)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696699 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902240936 (https://github.com/cluebotng/component-configs/commits/77ffbc8ebcebe0ba0b96808c0324623bd61aadf9)
* 15:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900617624 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898273109 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898254138 (https://github.com/cluebotng/component-configs/commits/5392cc3d04cf9e3086473ded50ae76eb8ffb6a29)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40894576159 (https://github.com/cluebotng/component-configs/commits/d180ba6310a8569d7c091af3c9dd92dcdbda1991)

=== 2026-10-28 ===
* 18:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217792 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe)
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507911 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788312 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987512 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879022 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)
* 02:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021123 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf)

=== 2026-10-27 ===
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112168 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846790 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 15:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588912 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106)
* 14:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783783 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 14:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784252 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108700 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591155 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231026 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21)
* 09:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261201 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249547 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601906 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed)
* 06:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40459540107 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798083 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 05:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40455721193 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 05:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518044 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826128 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9)
* 04:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765772 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375412 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527027 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444184085 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)
* 03:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895975 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736567 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6)

=== 2026-10-26 ===
* 23:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679243 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a)
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40378908239 (https://github.com/cluebotng/component-configs/commits/2d1531d1cbfbfa07a816f43b825ad9d8214f9996)

=== 2026-10-23 ===
* 22:56 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-09-03 ===
* 10:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33742087870 (https://github.com/cluebotng/component-configs/commits/781284a9a6b62d000c7c570fb34d20bbe21933a0)

=== 2026-09-01 ===
* 19:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856122 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3)
* 16:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531639211 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-25 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32866638493 (https://github.com/cluebotng/component-configs/commits/f0c0373d1612550e858604e6c40555d78d071318)
* 14:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32859995726 (https://github.com/cluebotng/component-configs/commits/8e54667ba485459bad6bd7798274c98ea487caee)

=== 2026-08-24 ===
* 09:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32713575575 (https://github.com/cluebotng/component-configs/commits/15461b10d7f124d327ec6065e11b3fce0465cc68)

=== 2026-07-30 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/30522988957 (https://github.com/cluebotng/component-configs/commits/e38d48bc554d39c0874c954226583b005318f76f)

=== 2026-07-23 ===
* 11:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/30003154188 (https://github.com/cluebotng/component-configs/commits/97520d86ff57924e8237a0446010f837e79b9062)
* 10:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29998195703 (https://github.com/cluebotng/component-configs/commits/e8762796572d00e69633a1b076413d5aa4ee3508)
* 09:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29995256262 (https://github.com/cluebotng/component-configs/commits/c08c3e19100546a5f8ed26a2bfa7095cb4633147)

=== 2026-07-20 ===
* 15:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29753268267 (https://github.com/cluebotng/component-configs/commits/3cd242ed54e9146f6a3dfd9305f97d4580c2fc06)
* 14:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29749326972 (https://github.com/cluebotng/component-configs/commits/83d05d6d10640a88d80bdbc041955ca27603a0fe)
* 13:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29744681881 (https://github.com/cluebotng/component-configs/commits/8b5c46ef715a2e57f383d35d7acc37bfbfa4db2d)
* 11:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29738720465 (https://github.com/cluebotng/component-configs/commits/661b4bae545b221090aa694fa30ff924415a5b22)
* 11:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29737812391 (https://github.com/cluebotng/component-configs/commits/10014b40131fb0e2f27f7de26102f6689287add4)

=== 2026-07-16 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29484727917 (https://github.com/cluebotng/component-configs/commits/328ca21897e26730c95c4dad77c34d5e1bbf9c31)
* 08:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29482938340 (https://github.com/cluebotng/component-configs/commits/6ddd234f9ddace81ab9e1e7e45b5f974c5938a11)

=== 2026-07-15 ===
* 17:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29436632270 (https://github.com/cluebotng/component-configs/commits/ff37e7e6d3e126ba24d186be144aeae3d2048954)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29434331433 (https://github.com/cluebotng/component-configs/commits/37dd33ec5b56f991e912ec44ce96a536735cb1a0)
* 16:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29430449675 (https://github.com/cluebotng/component-configs/commits/ecf4f749293c44e72e74858bef8193eb29695c9a)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29428509974 (https://github.com/cluebotng/component-configs/commits/842854f79b07dc4ec739505a99c8cb11009f8f4f)
* 15:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29427447334 (https://github.com/cluebotng/component-configs/commits/0d2bc5e6c57ba81a40584a85fe4ea3868875999f)
* 08:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29399091032 (https://github.com/cluebotng/component-configs/commits/5880b5b641021f572e9dd6a3d76443db4a607114)

=== 2026-07-14 ===
* 08:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29318761866 (https://github.com/cluebotng/component-configs/commits/04593c4734db7ed505ba34446adacc22176c18ed)

=== 2026-07-03 ===
* 13:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28663718696 (https://github.com/cluebotng/component-configs/commits/8b35b36bf642f83b59af6cf2903f9fcdd6d67007)
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28651139847 (https://github.com/cluebotng/component-configs/commits/11ad538e5876626229416e2ba16bc9f7a7e5cace)
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650658730 (https://github.com/cluebotng/component-configs/commits/0d3a15b63afd3afe5965be4754401712df0b4ae1)
* 09:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650009604 (https://github.com/cluebotng/component-configs/commits/a66ec11b207598051416135000b2a5815ba30080)

=== 2026-07-02 ===
* 22:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28625556671 (https://github.com/cluebotng/component-configs/commits/e95434d48b96490f3526640acde23ce59116589b)
* 21:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412770 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 20:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28618176241 (https://github.com/cluebotng/component-configs/commits/c3b2ebd135508a9a0f781ed332514069fb4d8c89)

=== 2026-07-01 ===
* 20:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28544657461 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861)
* 18:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28538748543 (https://github.com/cluebotng/component-configs/commits/6c7bcbaa2a10eb01c07a4df9fe5c1ea3b571b351)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28536333606 (https://github.com/cluebotng/component-configs/commits/fc7f0d0647c84e0e5c9f02d279015dc3183f3151)
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28525308526 (https://github.com/cluebotng/component-configs/commits/4d35560ba5b135c6dff161ca5a7ed6c393edd180)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506476 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53)

=== 2026-06-30 ===
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53)

=== 2026-06-29 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28404836827 (https://github.com/cluebotng/component-configs/commits/637e933186e3a7fedd0e7328d4c4e6035d12ec82)

=== 2026-06-27 ===
* 22:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337267 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 13:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28291090339 (https://github.com/cluebotng/component-configs/commits/feff80c249bc28b62ed154c74e8f8b1bfa597e3c)
* 13:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28291037948 (https://github.com/cluebotng/component-configs/commits/03f0e1e95886611fe4bf68c103a51f34ec7a8f41)

=== 2026-06-24 ===
* 18:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 17:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 16:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 14:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28103726388 (https://github.com/cluebotng/component-configs/commits/3ae736249e7b8cae3c57261c55b221a4d62296f9)

=== 2026-06-16 ===
* 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27639941201 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849976 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737553 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634022645 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770908 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 14:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179755 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-13 ===
* 13:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463956 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)

=== 2026-06-10 ===
* 15:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125849 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27279260980 (https://github.com/cluebotng/component-configs/commits/83e16c74dca286ed8f7104d49a271dee18c41854)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788838 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276699371 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510139 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411819 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397723 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56)
* 12:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339407 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225356702 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156804 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601458 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)

=== 2026-06-04 ===
* 22:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d)

=== 2026-06-01 ===
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978145 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757972034 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864800 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862186 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719590163 (https://github.com/cluebotng/component-configs/commits/9fc63fa520f0c2c3790d3d3236682dbb83382a9f)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575482 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d)

=== 2026-05-29 ===
* 00:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639432 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20)

=== 2026-05-28 ===
* 18:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792702 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8)

=== 2026-05-21 ===
* 20:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627543 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056)

=== 2026-05-19 ===
* 00:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076424 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c)

=== 2026-05-17 ===
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25986406214 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752565 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616046 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25727821548 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842111 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562306 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480578 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280492 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-07 ===
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25505647057 (https://github.com/cluebotng/component-configs/commits/874d7f6f407fc9a3995f52f40312cd7d3a712176)

=== 2026-05-06 ===
* 18:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754913 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-04 ===
* 07:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-05-02 ===
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278570 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)
* 00:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24865518502 (https://github.com/cluebotng/component-configs/commits/6e953f35fdca38226cde9ea7280f948f34242881)

=== 2026-04-23 ===
* 23:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24863962598 (https://github.com/cluebotng/component-configs/commits/07b9ad8616853af2ed49f96de6e55c14e3faabe0)
* 09:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24827343249 (https://github.com/cluebotng/component-configs/commits/218a7fa56222cf3e98b642eebbf6b1b2b273a92d)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826859245 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79)
* 09:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826656498 (https://github.com/cluebotng/component-configs/commits/90aa39b3e81f19f42d08d5ec6131ba09c68bd786)

=== 2026-04-17 ===
* 00:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24540678933 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24422836823 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442412 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24249897963 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247614873 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c)
* 14:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247570676 (https://github.com/cluebotng/component-configs/commits/93ced49392782bf65e34d13f10cbeaafa760f115)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093216 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656823 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659535101 (https://github.com/cluebotng/component-configs/commits/b825610c7f23870a8561b00b5f8546b107643015)

=== 2026-03-25 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729395 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640340 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551615 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381633193 (https://github.com/cluebotng/component-configs/commits/ac40c461942f4541b640a32ff0141268418abc12)

=== 2026-03-19 ===
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23315406679 (https://github.com/cluebotng/component-configs/commits/fd07020c08545c83ab35667616a26081966648df)

=== 2026-02-15 ===
* 10:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380291 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb)

=== 2025-11-11 ===
* 15:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301650 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266856634 (https://github.com/cluebotng/component-configs/commits/b0e9170597a778654185be762c580e2a6e19492f)

=== 2025-11-06 ===
* 01:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19121350626 (https://github.com/cluebotng/component-configs/commits/80f69f0ab7b09c2e3e5e208d847a954cb1975bc6)

=== 2025-11-05 ===
* 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995211 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9)
* 18:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19112506240 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)
* 16:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108852168 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217917 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872623 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267293 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-08 ===
* 15:47 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.1
* 11:23 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.0

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101448 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 09:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092350259 (https://github.com/cluebotng/component-configs/commits/283965c9240c0c5a72e0ea1203439583935295cb)

=== 2025-09-27 ===
* 13:07 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.3.0

=== 2025-09-26 ===
* 18:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649036 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca)
* 12:52 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.4
* 12:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18038096319 (https://github.com/cluebotng/component-configs/commits/e10e601b4fb06b3fd97856ef86a30e5391fb4f17)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926507 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d)
* 12:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037156332 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)

=== 2025-09-25 ===
* 21:20 wmbot~damian@tools-bastion-15: bot deployed @ v1.2.2
* 21:20 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.2
* 19:17 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.1
* 19:16 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.0
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18017003184 (https://github.com/cluebotng/component-configs/commits/e9fc8d46ac1a0ff0ac6203458fa171c6430492ce)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016894528 (https://github.com/cluebotng/component-configs/commits/cfc9adc9516df0f11c8b6d1df68232d0a46cb4eb)
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998407 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 17:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801881 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221965 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782359 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677281 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531091 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)

=== 2025-09-22 ===
* 19:08 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.3
* 19:04 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.2
* 19:02 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.1

=== 2025-08-29 ===
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.2.0
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.1

=== 2025-08-15 ===
* 21:14 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.0
* 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/1.0.34
* 12:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.33
* 00:23 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.32
* 00:08 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.31

=== 2025-08-14 ===
* 16:13 wmbot~damian@tools-bastion-13: irc-relay deployed @ v1.1.12
* 12:47 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2
* 12:43 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2

=== 2025-08-11 ===
* 13:06 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.3
* 12:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.30

=== 2025-08-10 ===
* 17:42 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.2
* 17:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.29
* 16:35 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.1

=== 2025-08-08 ===
* 15:37 wmbot~damian@tools-bastion-12: Updated ci-execute-fabric to use dedicated unix account, dropped key from human account
* 15:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.28
* 15:33 wmbot~damian@tools-bastion-13: report deployed @ refs/tags/v1.0.28

=== 2025-08-07 ===
* 15:24 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27

=== 2024-03-21 ===
* 10:13 dcaro: fixed .lighttpd.conf file to add port and remove socket

=== 2021-07-23 ===
* 16:40 majavah: stop cbng_relay grid job, still having issues with irc connection - [[phab:T274871|T274871]]

=== 2020-01-13 ===
* 20:53 wm-bot: &lt;root&gt; Restarted webservice to fix broken registration with the front proxy ([[phab:T242538|T242538]])

=== 2019-03-07 ===
* 05:06 bd808: Killed cbng_bot job stuck in deletion state with 4000+ zombie child processes ([[phab:T217817|T217817]])


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>kkatwlf2o78hpmh6brfgxiysb1awz5m</sha1>
    </revision>
    <revision>
      <id>2478999</id>
      <parentid>2478989</parentid>
      <timestamp>2026-11-07T12:44:05Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373224062 (https://github.com/cluebotng/component-configs/commits/b7987054470c43be35c9e93b1e91e43119048538)</comment>
      <origin>2478999</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="54545" sha1="otkh2jbk55nrtk4pmuyw872oeflrp52" xml:space="preserve">=== 2026-11-07 ===
* 12:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373224062 (https://github.com/cluebotng/component-configs/commits/b7987054470c43be35c9e93b1e91e43119048538)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856570 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819762 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251009 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 16:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695823 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249636648 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248508521 (https://github.com/cluebotng/component-configs/commits/3acae96966e16f4f9f91683ad36f5e56d329753a)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234031825 (https://github.com/cluebotng/component-configs/commits/15c352e208f001eee46b8462013282f7e93d9872)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233327546 (https://github.com/cluebotng/component-configs/commits/e9471c88503e10f4ba84992f33bfa015a2c6c887)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248920 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188059 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 13:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42222101673 (https://github.com/cluebotng/component-configs/commits/5a9290a457af2e5700d955bc65fbd8d77cad712c)
* 12:21 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485438 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 10:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42200307462 (https://github.com/cluebotng/component-configs/commits/255eec1d87ca571dcfd0f09253a9504e7d2b70f1)
* 07:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181945119 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270625 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297609 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42166677976 (https://github.com/cluebotng/component-configs/commits/ceeb81f4d5e5d7b6a4947b95df020093bdea2903)
* 04:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42162423731 (https://github.com/cluebotng/component-configs/commits/f0d701fcec7061cd6d2bf7ea127a73895e4a005f)
* 03:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42155618053 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 05:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41976470110 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974543482 (https://github.com/cluebotng/component-configs/commits/7f5f130ad27cc6b58d18dc7db9fb6f17a994795f)
* 05:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300376 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 04:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41970293977 (https://github.com/cluebotng/component-configs/commits/249d7f9d3e64c31431d63973b2e535b32af4e788)
* 03:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41961409161 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 23:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942732 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214126 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107372 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a)
* 04:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41773053080 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208829 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274447 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064641 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2)
* 00:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119511 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580770 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7)
* 12:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616293 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215687 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b)
* 05:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264002 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147096 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696228 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e)
* 02:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216287 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434850 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-11-01 ===
* 21:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41350442746 (https://github.com/cluebotng/component-configs/commits/9f8d02aa99dd4b425755d1cf842fbce1fa045266)
* 20:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41346673895 (https://github.com/cluebotng/component-configs/commits/eea2ac3e027177d39bf8015524cc6b02f32445f9)
* 12:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41306290063 (https://github.com/cluebotng/component-configs/commits/d2705aba98618c9fa1aecc8578e6861ed2ef9ef8)

=== 2026-10-31 ===
* 16:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41219859885 (https://github.com/cluebotng/component-configs/commits/ade6a3988f2a8e1609e94a58b7cb546484dd8e9a)
* 14:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074802 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347107 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603041 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191380192 (https://github.com/cluebotng/component-configs/commits/2be620aa8d34f581c9c9bb72401e2bcb82705c7d)
* 09:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726845 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 09:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186724198 (https://github.com/cluebotng/component-configs/commits/8474642453be7b488acccbcd8e7bfe5261236e6c)
* 08:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002299 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139892801 (https://github.com/cluebotng/component-configs/commits/ce1c920d2a60015846167ad5fba607bfa99e27fa)
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595319 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610828 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626173 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41135430856 (https://github.com/cluebotng/component-configs/commits/118e964d982be74b124949a21f67c966a2b2bea0)
* 21:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41131252248 (https://github.com/cluebotng/component-configs/commits/9aadd326a870be6402adf95e5254a733fdc7c0ac)
* 21:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558288 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761927 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)

=== 2026-10-29 ===
* 15:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40903315370 (https://github.com/cluebotng/component-configs/commits/48d12f535986ffcf04bdc857a204667a10e4579d)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696699 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902240936 (https://github.com/cluebotng/component-configs/commits/77ffbc8ebcebe0ba0b96808c0324623bd61aadf9)
* 15:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900617624 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898273109 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898254138 (https://github.com/cluebotng/component-configs/commits/5392cc3d04cf9e3086473ded50ae76eb8ffb6a29)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40894576159 (https://github.com/cluebotng/component-configs/commits/d180ba6310a8569d7c091af3c9dd92dcdbda1991)

=== 2026-10-28 ===
* 18:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217792 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe)
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507911 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788312 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987512 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879022 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)
* 02:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021123 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf)

=== 2026-10-27 ===
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112168 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846790 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 15:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588912 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106)
* 14:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783783 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 14:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784252 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108700 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591155 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231026 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21)
* 09:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261201 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249547 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601906 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed)
* 06:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40459540107 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798083 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 05:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40455721193 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 05:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518044 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826128 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9)
* 04:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765772 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375412 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527027 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444184085 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)
* 03:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895975 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736567 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6)

=== 2026-10-26 ===
* 23:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679243 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a)
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40378908239 (https://github.com/cluebotng/component-configs/commits/2d1531d1cbfbfa07a816f43b825ad9d8214f9996)

=== 2026-10-23 ===
* 22:56 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-09-03 ===
* 10:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33742087870 (https://github.com/cluebotng/component-configs/commits/781284a9a6b62d000c7c570fb34d20bbe21933a0)

=== 2026-09-01 ===
* 19:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856122 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3)
* 16:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531639211 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-25 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32866638493 (https://github.com/cluebotng/component-configs/commits/f0c0373d1612550e858604e6c40555d78d071318)
* 14:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32859995726 (https://github.com/cluebotng/component-configs/commits/8e54667ba485459bad6bd7798274c98ea487caee)

=== 2026-08-24 ===
* 09:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/32713575575 (https://github.com/cluebotng/component-configs/commits/15461b10d7f124d327ec6065e11b3fce0465cc68)

=== 2026-07-30 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/30522988957 (https://github.com/cluebotng/component-configs/commits/e38d48bc554d39c0874c954226583b005318f76f)

=== 2026-07-23 ===
* 11:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/30003154188 (https://github.com/cluebotng/component-configs/commits/97520d86ff57924e8237a0446010f837e79b9062)
* 10:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29998195703 (https://github.com/cluebotng/component-configs/commits/e8762796572d00e69633a1b076413d5aa4ee3508)
* 09:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29995256262 (https://github.com/cluebotng/component-configs/commits/c08c3e19100546a5f8ed26a2bfa7095cb4633147)

=== 2026-07-20 ===
* 15:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29753268267 (https://github.com/cluebotng/component-configs/commits/3cd242ed54e9146f6a3dfd9305f97d4580c2fc06)
* 14:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29749326972 (https://github.com/cluebotng/component-configs/commits/83d05d6d10640a88d80bdbc041955ca27603a0fe)
* 13:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29744681881 (https://github.com/cluebotng/component-configs/commits/8b5c46ef715a2e57f383d35d7acc37bfbfa4db2d)
* 11:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29738720465 (https://github.com/cluebotng/component-configs/commits/661b4bae545b221090aa694fa30ff924415a5b22)
* 11:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29737812391 (https://github.com/cluebotng/component-configs/commits/10014b40131fb0e2f27f7de26102f6689287add4)

=== 2026-07-16 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29484727917 (https://github.com/cluebotng/component-configs/commits/328ca21897e26730c95c4dad77c34d5e1bbf9c31)
* 08:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29482938340 (https://github.com/cluebotng/component-configs/commits/6ddd234f9ddace81ab9e1e7e45b5f974c5938a11)

=== 2026-07-15 ===
* 17:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29436632270 (https://github.com/cluebotng/component-configs/commits/ff37e7e6d3e126ba24d186be144aeae3d2048954)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29434331433 (https://github.com/cluebotng/component-configs/commits/37dd33ec5b56f991e912ec44ce96a536735cb1a0)
* 16:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29430449675 (https://github.com/cluebotng/component-configs/commits/ecf4f749293c44e72e74858bef8193eb29695c9a)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29428509974 (https://github.com/cluebotng/component-configs/commits/842854f79b07dc4ec739505a99c8cb11009f8f4f)
* 15:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29427447334 (https://github.com/cluebotng/component-configs/commits/0d2bc5e6c57ba81a40584a85fe4ea3868875999f)
* 08:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29399091032 (https://github.com/cluebotng/component-configs/commits/5880b5b641021f572e9dd6a3d76443db4a607114)

=== 2026-07-14 ===
* 08:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29318761866 (https://github.com/cluebotng/component-configs/commits/04593c4734db7ed505ba34446adacc22176c18ed)

=== 2026-07-03 ===
* 13:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28663718696 (https://github.com/cluebotng/component-configs/commits/8b35b36bf642f83b59af6cf2903f9fcdd6d67007)
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28651139847 (https://github.com/cluebotng/component-configs/commits/11ad538e5876626229416e2ba16bc9f7a7e5cace)
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650658730 (https://github.com/cluebotng/component-configs/commits/0d3a15b63afd3afe5965be4754401712df0b4ae1)
* 09:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28650009604 (https://github.com/cluebotng/component-configs/commits/a66ec11b207598051416135000b2a5815ba30080)

=== 2026-07-02 ===
* 22:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28625556671 (https://github.com/cluebotng/component-configs/commits/e95434d48b96490f3526640acde23ce59116589b)
* 21:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412770 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 20:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28618176241 (https://github.com/cluebotng/component-configs/commits/c3b2ebd135508a9a0f781ed332514069fb4d8c89)

=== 2026-07-01 ===
* 20:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28544657461 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861)
* 18:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28538748543 (https://github.com/cluebotng/component-configs/commits/6c7bcbaa2a10eb01c07a4df9fe5c1ea3b571b351)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28536333606 (https://github.com/cluebotng/component-configs/commits/fc7f0d0647c84e0e5c9f02d279015dc3183f3151)
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28525308526 (https://github.com/cluebotng/component-configs/commits/4d35560ba5b135c6dff161ca5a7ed6c393edd180)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506476 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53)

=== 2026-06-30 ===
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53)

=== 2026-06-29 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28404836827 (https://github.com/cluebotng/component-configs/commits/637e933186e3a7fedd0e7328d4c4e6035d12ec82)

=== 2026-06-27 ===
* 22:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337267 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 13:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28291090339 (https://github.com/cluebotng/component-configs/commits/feff80c249bc28b62ed154c74e8f8b1bfa597e3c)
* 13:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28291037948 (https://github.com/cluebotng/component-configs/commits/03f0e1e95886611fe4bf68c103a51f34ec7a8f41)

=== 2026-06-24 ===
* 18:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117355708 (https://github.com/cluebotng/component-configs/commits/7724999a3b81d08da9839a618cfc82ac0b2e156b)
* 17:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 17:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 16:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28114545973 (https://github.com/cluebotng/component-configs/commits/ac72be63d926548f9b3b5bd424908a3219a6cc01)
* 14:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28103726388 (https://github.com/cluebotng/component-configs/commits/3ae736249e7b8cae3c57261c55b221a4d62296f9)

=== 2026-06-16 ===
* 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27639941201 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849976 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737553 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634022645 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770908 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 14:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179755 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-13 ===
* 13:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463956 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)

=== 2026-06-10 ===
* 15:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125849 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27279260980 (https://github.com/cluebotng/component-configs/commits/83e16c74dca286ed8f7104d49a271dee18c41854)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788838 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276699371 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510139 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411819 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397723 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56)
* 12:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339407 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225356702 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156804 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601458 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)

=== 2026-06-04 ===
* 22:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d)

=== 2026-06-01 ===
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978145 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757972034 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864800 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862186 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719590163 (https://github.com/cluebotng/component-configs/commits/9fc63fa520f0c2c3790d3d3236682dbb83382a9f)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575482 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d)

=== 2026-05-29 ===
* 00:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639432 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20)

=== 2026-05-28 ===
* 18:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792702 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8)

=== 2026-05-21 ===
* 20:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627543 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056)

=== 2026-05-19 ===
* 00:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076424 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c)

=== 2026-05-17 ===
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25986406214 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752565 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616046 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25727821548 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842111 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562306 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480578 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280492 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-07 ===
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25505647057 (https://github.com/cluebotng/component-configs/commits/874d7f6f407fc9a3995f52f40312cd7d3a712176)

=== 2026-05-06 ===
* 18:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754913 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-04 ===
* 07:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-05-02 ===
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278570 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)
* 00:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24865518502 (https://github.com/cluebotng/component-configs/commits/6e953f35fdca38226cde9ea7280f948f34242881)

=== 2026-04-23 ===
* 23:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24863962598 (https://github.com/cluebotng/component-configs/commits/07b9ad8616853af2ed49f96de6e55c14e3faabe0)
* 09:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24827343249 (https://github.com/cluebotng/component-configs/commits/218a7fa56222cf3e98b642eebbf6b1b2b273a92d)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826859245 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79)
* 09:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826656498 (https://github.com/cluebotng/component-configs/commits/90aa39b3e81f19f42d08d5ec6131ba09c68bd786)

=== 2026-04-17 ===
* 00:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24540678933 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24422836823 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442412 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24249897963 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247614873 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c)
* 14:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247570676 (https://github.com/cluebotng/component-configs/commits/93ced49392782bf65e34d13f10cbeaafa760f115)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093216 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656823 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659535101 (https://github.com/cluebotng/component-configs/commits/b825610c7f23870a8561b00b5f8546b107643015)

=== 2026-03-25 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729395 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640340 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551615 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381633193 (https://github.com/cluebotng/component-configs/commits/ac40c461942f4541b640a32ff0141268418abc12)

=== 2026-03-19 ===
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23315406679 (https://github.com/cluebotng/component-configs/commits/fd07020c08545c83ab35667616a26081966648df)

=== 2026-02-15 ===
* 10:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380291 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb)

=== 2025-11-11 ===
* 15:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301650 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266856634 (https://github.com/cluebotng/component-configs/commits/b0e9170597a778654185be762c580e2a6e19492f)

=== 2025-11-06 ===
* 01:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19121350626 (https://github.com/cluebotng/component-configs/commits/80f69f0ab7b09c2e3e5e208d847a954cb1975bc6)

=== 2025-11-05 ===
* 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995211 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9)
* 18:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19112506240 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)
* 16:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108852168 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217917 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872623 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267293 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-08 ===
* 15:47 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.1
* 11:23 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.0

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101448 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 09:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092350259 (https://github.com/cluebotng/component-configs/commits/283965c9240c0c5a72e0ea1203439583935295cb)

=== 2025-09-27 ===
* 13:07 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.3.0

=== 2025-09-26 ===
* 18:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649036 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca)
* 12:52 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.4
* 12:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18038096319 (https://github.com/cluebotng/component-configs/commits/e10e601b4fb06b3fd97856ef86a30e5391fb4f17)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926507 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d)
* 12:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037156332 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)

=== 2025-09-25 ===
* 21:20 wmbot~damian@tools-bastion-15: bot deployed @ v1.2.2
* 21:20 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.2
* 19:17 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.1
* 19:16 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.0
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18017003184 (https://github.com/cluebotng/component-configs/commits/e9fc8d46ac1a0ff0ac6203458fa171c6430492ce)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016894528 (https://github.com/cluebotng/component-configs/commits/cfc9adc9516df0f11c8b6d1df68232d0a46cb4eb)
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998407 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 17:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801881 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221965 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782359 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677281 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531091 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)

=== 2025-09-22 ===
* 19:08 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.3
* 19:04 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.2
* 19:02 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.1

=== 2025-08-29 ===
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.2.0
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.1

=== 2025-08-15 ===
* 21:14 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.0
* 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/1.0.34
* 12:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.33
* 00:23 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.32
* 00:08 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.31

=== 2025-08-14 ===
* 16:13 wmbot~damian@tools-bastion-13: irc-relay deployed @ v1.1.12
* 12:47 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2
* 12:43 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2

=== 2025-08-11 ===
* 13:06 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.3
* 12:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.30

=== 2025-08-10 ===
* 17:42 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.2
* 17:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.29
* 16:35 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.1

=== 2025-08-08 ===
* 15:37 wmbot~damian@tools-bastion-12: Updated ci-execute-fabric to use dedicated unix account, dropped key from human account
* 15:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.28
* 15:33 wmbot~damian@tools-bastion-13: report deployed @ refs/tags/v1.0.28

=== 2025-08-07 ===
* 15:24 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27

=== 2024-03-21 ===
* 10:13 dcaro: fixed .lighttpd.conf file to add port and remove socket

=== 2021-07-23 ===
* 16:40 majavah: stop cbng_relay grid job, still having issues with irc connection - [[phab:T274871|T274871]]

=== 2020-01-13 ===
* 20:53 wm-bot: &lt;root&gt; Restarted webservice to fix broken registration with the front proxy ([[phab:T242538|T242538]])

=== 2019-03-07 ===
* 05:06 bd808: Killed cbng_bot job stuck in deletion state with 4000+ zombie child processes ([[phab:T217817|T217817]])


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>otkh2jbk55nrtk4pmuyw872oeflrp52</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.wd-image-positions/SAL</title>
    <ns>498</ns>
    <id>443947</id>
    <revision>
      <id>2478972</id>
      <parentid>2478965</parentid>
      <timestamp>2026-11-07T12:09:24Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~lucaswerkmeister@tools-bastion-15: deployed ea680c0: Highlighted depicted matching iiif_region in URL</comment>
      <origin>2478972</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="29849" sha1="ns1kfxtjxez7holyimyhkbt80hoe18y" xml:space="preserve">=== 2026-11-07 ===
* 12:09 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ea680c0}}: Highlighted depicted matching iiif_region in URL
* 11:41 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|eb10545}}: Add support for querying WDQS v2 (query-next)
* 11:39 wmbot~lucaswerkmeister@tools-bastion-15: toolforge envvars create TOOL_QUERY_NEXT_USER_AGENT

=== 2026-10-31 ===
* 14:33 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|11cad2b}}: Enable Toolforge alerts

=== 2026-10-22 ===
* 18:10 lucaswerkmeister: deployed {{Gerrit|d69d109}}: Upgrade dependencies

=== 2026-10-19 ===
* 18:08 wmbot~lucaswerkmeister@tools-bastion-15: deployed 9ac1608..f23674a: l10n updates (skr-arab)

=== 2026-10-08 ===
* 19:38 wmbot~lucaswerkmeister@tools-bastion-15: deployed 2578960..9ac1608: l10n updates (mag)

=== 2026-09-22 ===
* 20:18 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b5a93138aa}} (Python 3.14)
* 20:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c00bcb6310}} (push-to-deploy) and {{Gerrit|550f1df480}} (upgrade dependencies)

=== 2026-09-21 ===
* 12:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|45d853810e}} (l10n updates: pl)

=== 2026-08-31 ===
* 17:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e0465e8dfe}} (l10n updates: pl) [actually deployed this time, oops]
* 17:53 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e0465e8dfe}} (l10n updates: pl)

=== 2026-08-27 ===
* 19:38 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|42c2642f71}} (l10n updates: pl)

=== 2026-08-23 ===
* 12:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|6b1c827261}} (l10n updates: cs, ha, pl)

=== 2026-07-22 ===
* 10:37 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e33530c9d2}} (fix README)

=== 2026-07-02 ===
* 19:33 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|bb43718fe8}} (l10n updates: kw)

=== 2026-06-25 ===
* 14:53 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b3412fb959}} (l10n updates: cs)

=== 2026-06-08 ===
* 12:40 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1d78cb4fd9}} (l10n updates: nl)

=== 2026-05-25 ===
* 13:50 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2d3ce61727}} (l10n updates: nl)

=== 2026-05-14 ===
* 14:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5e96e1a7f9}} (l10n updates: ko, vi)

=== 2026-04-20 ===
* 12:57 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|135b3fa3d2}} (l10n updates: ko)

=== 2026-04-13 ===
* 17:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|9c45de24ca}} (l10n updates: hi)

=== 2026-04-09 ===
* 12:30 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|141811e50d}} (l10n updates: ru)

=== 2026-04-06 ===
* 12:36 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5f19f74dbf}} (l10n updates: ms-arab)

=== 2026-03-30 ===
* 12:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7b4b75736e}} (l10n updates: ja, ru)

=== 2026-03-23 ===
* 19:01 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|14f05f96fc}} (l10n updates: kea)

=== 2026-03-19 ===
* 13:17 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|6ee3f33c6e}} (l10n updates: kea, ps)

=== 2026-03-17 ===
* 17:20 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3f7e40bfcd}} (fix [[phab:T420332|T420332]])

=== 2026-03-16 ===
* 13:20 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d56726dd72}} (l10n updates: ga)

=== 2026-03-12 ===
* 16:31 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|dd76898d14}} (l10n updates: kea)

=== 2026-03-09 ===
* 13:21 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c3f6d784d6}} (l10n updates: kea, wal)

=== 2026-03-03 ===
* 17:21 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe77d87b89}} (srcset= improvements, including standard image widths [[phab:T414805|T414805]])

=== 2026-02-23 ===
* 20:41 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|e1fb92b22a}} (l10n updates: cs)

=== 2026-01-29 ===
* 18:00 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|a644f69164}} (l10n updates: pt-br)

=== 2026-01-03 ===
* 17:01 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f5b6d0848c}} (l10n updates: ca, it, kab, sv)

=== 2025-12-26 ===
* 09:09 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0fc65339ab}} (l10n updates: el, it)

=== 2025-12-22 ===
* 19:06 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d64c87fd74}} (l10n updates: it)

=== 2025-12-08 ===
* 22:02 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|09b3779571}} (l10n updates: tg-cyrl)

=== 2025-11-27 ===
* 12:57 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|1aecb0fe93}} (l10n updates: tg-cyrl)

=== 2025-11-18 ===
* 20:05 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2c255742e4}} (fix skiplink visibility)

=== 2025-11-10 ===
* 19:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|91b8735f98}} (upgrade dependencies)
* 18:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|13e06edac8}} (l10n updates: frp, ro)

=== 2025-11-03 ===
* 19:26 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|45b1f77536}} (l10n updates: hu)

=== 2025-10-30 ===
* 20:07 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ba00c92428}} (l10n updates: tt-cyrl)

=== 2025-10-27 ===
* 14:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|8b50c8aff5}} (l10n updates: eo, ru, tg-cyrl, yue-hant)

=== 2025-10-20 ===
* 12:24 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f292fb9642}} (l10n updates: frp)

=== 2025-10-16 ===
* 19:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c4f90b172d}} (l10n updates: frp)

=== 2025-10-13 ===
* 18:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|041f0cc802}} (l10n updates: ru)

=== 2025-09-29 ===
* 18:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|cdcf00a3ba}} (l10n updates: ko-kp)

=== 2025-09-18 ===
* 20:08 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fcd38fc5a4}} (l10n updates: rki, ru)

=== 2025-08-25 ===
* 17:07 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c8209d189b}} (l10n updates: aig, pt)

=== 2025-08-24 ===
* 22:59 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|80d93db69e}} (l10n updates: el, pt, pt-br, ru, sl, yue-hant)
* 22:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|dbd4513115}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility)

=== 2025-07-23 ===
* 11:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9460de6e30}} (Codex 2.2.0)

=== 2025-07-15 ===
* 17:51 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|256e149618}} (fix dev-requirements.in)

=== 2025-07-13 ===
* 15:39 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|168a4bf7cc}} (upgrade m3api)
* 15:31 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ebfaeef6e0}} (specify .python-version)
* 15:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|81a627821b}} (Python 3.13 + Toolforge Build Service)
* 15:27 lucaswerkmeister: webservice stop &amp;&amp; mv www&lt;nowiki&gt;{&lt;/nowiki&gt;,-unused-tool-now-runs-on-buildservice&lt;nowiki&gt;}&lt;/nowiki&gt; &amp;&amp; wget https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/raw/81a627821b/service.template &amp;&amp; webservice start
* 14:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b27c4c2d73}} (read config from envvars)
* 14:48 wmbot~lucaswerkmeister@tools-bastion-13: commented out config.yaml, should use envvars instead
* 14:47 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;SECRET_KEY&quot;]))' {{!}} toolforge envvars create TOOL_SECRET_KEY
* 14:46 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;OAUTH&quot;][&quot;CONSUMER_SECRET&quot;]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET
* 14:46 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;OAUTH&quot;][&quot;CONSUMER_KEY&quot;]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY
* 14:43 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b0af29d932}} (change config keys to uppercase to work around [[phab:T374780|T374780]])
* 13:51 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|451794996c}} (add health-check-path)
* 13:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|99489081d3}} (split dev requirements from prod requirements)
* 12:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d9d2273efb}} (upgrade dependencies)

=== 2025-07-10 ===
* 19:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b2bc6237f0}} (l10n updates: hu)

=== 2025-07-07 ===
* 16:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8ff73093a1}} (l10n updates: uk)
* 06:31 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d1f4beb35e}} (l10n updates: af, ru, tr)

=== 2025-06-11 ===
* 22:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9d8920826c}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]])

=== 2025-05-26 ===
* 16:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9cea85d255}} (l10n updates: tr)

=== 2025-05-15 ===
* 21:51 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|17a80202ae}} (l10n updates: it)

=== 2025-05-08 ===
* 13:07 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a330358f39}} (l10n updates: hu, ka); also includes {{Gerrit|8865bb0c67}} (l10n updates: kaa) – apparently I forgot to `git rebase` after `git fetch` last time 🤦

=== 2025-04-24 ===
* 19:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8865bb0c67}} (l10n updates: kaa)

=== 2025-04-21 ===
* 18:14 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|840b383c90}} (upgrade dependencies, including Flask 3.1.0 and toolforge-i18n 0.1.2)
* 16:46 wmbot~lucaswerkmeister@tools-bastion-13: webservice restart # clear out a stuck broken variable in toolforge-i18n

=== 2025-04-17 ===
* 19:04 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|919afec938}} (l10n updates: es)

=== 2025-04-14 ===
* 18:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0b184b8268}} (l10n updates: af)

=== 2025-03-07 ===
* 19:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b8022cddca}} (l10n updates: tr)

=== 2025-02-27 ===
* 12:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f053d010c6}} (l10n updates: sah)

=== 2025-02-13 ===
* 21:39 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2f4a78c0ad}} (l10n updates: rki)

=== 2025-01-30 ===
* 20:55 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|223cf553fb}} (l10n updates: ar)

=== 2025-01-09 ===
* 12:45 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6e55549156}} (l10n updates: lb)

=== 2025-01-06 ===
* 20:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b7c6f2e7e5}} (l10n updates: cs, sv)

=== 2024-12-16 ===
* 20:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5bea039bac}} (l10n updates: bbc-latn, lb)

=== 2024-11-28 ===
* 21:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|602dbded5b}} (l10n updates: tcy)

=== 2024-11-21 ===
* 22:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3486d60faf}} (l10n updates: tcy)

=== 2024-11-18 ===
* 19:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bfcbf544da}} (remove canonical from service.template)
* 19:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1527c5d7c8}} (l10n updates: krc)

=== 2024-11-11 ===
* 19:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a6d0e2ad43}} (l10n updates: ar)

=== 2024-11-05 ===
* 18:30 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|33511f4de7}} (l10n updates: it, lb, sr-ec, zh-hans, zh-hant; had been blocked by [[phab:T373807|T373807]])

=== 2024-10-25 ===
* 19:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4bf78a4c42}} (upgrade dependencies, including Werkzeug 3.0.6)

=== 2024-10-13 ===
* 11:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a4d0774ea2}} (upgrade dependencies, including MarkupSafe 3.0); also deploys {{Gerrit|f6af6d59e6}} (l10n updates: ar, ca, el, es, it, sv, uk; had been broken for a while due to [[phab:T373807|T373807]]) which I previously accidentally “deployed” and logged in quickcategories instead, oops :D

=== 2024-10-03 ===
* 16:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|694704b8ab}} (upgrade dependencies, including toolforge_i18n and Werkzeug 3.0.4)

=== 2024-09-01 ===
* 14:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6430a15668}} (l10n updates: qqq fix)

=== 2024-08-05 ===
* 13:35 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|898795588f}} (upgrade toolforge_i18n to 0.0.7)

=== 2024-08-01 ===
* 18:39 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|95846f654a}} (l10n updates: zh-hans)

=== 2024-07-31 ===
* 19:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5540ef17c9}} (upgrade toolforge_i18n to 0.0.6; also upgrade pip to 24.2)

=== 2024-07-30 ===
* 17:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3f5ba1a732}} (l10n updates: wal)

=== 2024-07-25 ===
* 19:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f515834cff}} (l10n updates: de)

=== 2024-07-22 ===
* 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|54ad2210a2}} (l10n updates: ar, kaa)

=== 2024-07-21 ===
* 18:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a291d8f4ab}} (upgrade toolforge_i18n to 0.0.5; also upgrade pip to 24.1.2)

=== 2024-07-19 ===
* 17:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4757a1f91e}} (l100n updates: de, nb, uk)

=== 2024-07-08 ===
* 18:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ee1bf2fdeb}} (l10n updates: ms, sk)

=== 2024-07-07 ===
* 18:13 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|01d9085f1a}} (upgrade toolforge_i18n to 0.0.2; also upgrade pip from 24.0 to 24.1.1)

=== 2024-07-01 ===
* 17:09 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|64c8192524}} (l10n updates: nb)

=== 2024-06-17 ===
* 16:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1b8e7b6b05}} (l10n updates: it, nl)

=== 2024-06-15 ===
* 12:36 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|887589b017}} (install toolforge_i18n from PyPI)

=== 2024-06-14 ===
* 17:51 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|62f5575dbd}} (fix logout, noticed while testing [[phab:T367188|T367188]])
* 17:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|606e0ca438}} ([[phab:T367188|T367188]])

=== 2024-06-10 ===
* 18:37 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|dcf0c5040e}} (l10n updates: ru)

=== 2024-06-07 ===
* 09:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|88402067dd}} (update GitHub actions), pulled without webservice restart
* 09:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|24a82d8701}} (search items in interface language)

=== 2024-06-06 ===
* 20:23 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ea1e8c10e2}} (fix language fallback in edit interface)
* 20:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|72a68851e0}} (fix TIFF)
* 19:55 wmbot~lucaswerkmeister@tools-bastion-13: webservice restart # was unresponsive, idk why

=== 2024-06-03 ===
* 12:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|aa319bc391}} (l10n updates: fr, he, qqq)

=== 2024-05-30 ===
* 17:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|53074960ca}} (l10n updates: es, fr)

=== 2024-05-27 ===
* 17:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|40f6405ab3}} (l10n updates: fr, ko, ru)

=== 2024-05-23 ===
* 20:10 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|b82dc1062f}} (l10n updates: de, gl, ko, lb, mk, sl, sr-ec)

=== 2024-05-20 ===
* 13:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2965e37ca2}} (l10n updates: fr, ia, it, ms, sr-ec, zh-hans, zh-hant)

=== 2024-05-19 ===
* 16:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0486d8ade3}} (make rest of tool translatable; resolves [[phab:T363626|T363626]])
* 14:46 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|903c13a18a}} (make editing interface translatable, [[phab:T363626|T363626]])
* 13:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bdd23c3081}} (fix template)
* 13:39 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bd8f091ad1}} (make “… with no region specified” translatable, [[phab:T363626|T363626]])

=== 2024-05-18 ===
* 14:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|108e8a5fa6}} (fix csrf_token check)

=== 2024-05-16 ===
* 20:08 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8f374ee202}} (l10n updates: es, zh-hans)

=== 2024-05-13 ===
* 18:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1ca9f6af8c}} (l10n updates: gl, he, ia, lb, mk, nl, sl, sr-ec, zh-hant)

=== 2024-05-09 ===
* 15:11 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e8d47d1cb7}} (improve toolforge_i18n and upgrade dependencies for newer Babel)

=== 2024-05-06 ===
* 20:14 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e5e9116fc9}} (optimize GitLab CI; pulled without webservice restart)
* 18:27 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e8d438134b}} (add GitLab CI for [[phab:T363626|T363626]]; pulled without webservice restart)
* 15:45 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8b677d25a9}} (README update, pulled without webservice restart)
* 15:12 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e4205d9cee}} (l10n updates: de, es, fr, it, zh-hant)

=== 2024-05-05 ===
* 10:47 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|6b0e493e20}} (fix language setting, [[phab:T363626|T363626]])
* 10:42 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|73baa98186}} (l10n updates: de, eu, fi, ko, mk, nb, nl; [[phab:T363626|T363626]])
* 09:53 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c6407a31ae}} (make “logged in as” translatable, [[phab:T363626|T363626]])
* 09:48 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|17817c0cad}} (GitHub actions fix, pulled without webservice restart, [[phab:T363626|T363626]])
* 09:41 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|279399cf63}} (fix double “image scale” text from language fallback)
* 09:35 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|4da2a2bc42}} (settings page, [[phab:T363626|T363626]])

=== 2024-05-04 ===
* 12:24 wmbot~lucaswerkmeister@tools-sgebastion-10: added l10n-bot as developer member on GitLab (ca. 30 minutes ago, but logging now for the record) ([[phab:T363626|T363626]])
* 12:24 wmbot~lucaswerkmeister@tools-sgebastion-10: added l10n-bot as developer member on GitLab (ca. 30 minutes ago, but logging now for the record)
* 12:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|418ca66477}} (make translatable using toolforge_i18n: [[phab:T363626|T363626]])

=== 2024-05-03 ===
* 17:15 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|3204d71b37}} (upgrade dependencies for Python 3.12 compat; also upgraded pip&lt;nowiki&gt;{&lt;/nowiki&gt;,-tools&lt;nowiki&gt;}&lt;/nowiki&gt; and wheel while I’m at it)

=== 2024-04-08 ===
* 18:06 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|f6dfc23eec}} (make session permanent after login) + {{Gerrit|7776444690}} (use property labels on index page)

=== 2023-12-11 ===
* 01:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bf2f609917}} (fix Vue template)

=== 2023-10-25 ===
* 20:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7aec0c0b10}} (use Codex 1.0.0)
* 18:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f7a426b0de}} (Werkzeug 3.0.1)

=== 2023-10-03 ===
* 10:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9d55c80b99}} (update dependencies, Flask+Werkzeug 3) [actually ~10 mins ago but I forgot to log it]

=== 2023-07-23 ===
* 15:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|143b86e1b3}} (disable autoCrop)
* 15:35 wm-bot: &lt;lucaswerkmeister&gt; *actually* deployed {{Gerrit|a7c924557b}} (forgot to git rebase)
* 13:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7c924557b}} (fix color of newly added regions)
* 13:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a479974a9d}} (only load edit interface when logged in)

=== 2023-07-19 ===
* 18:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0e0deb1a49}} (reimplement “add new depicted” form in Codex)

=== 2023-07-15 ===
* 10:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|81999c31af}} (Python 3.11)

=== 2023-05-01 ===
* 17:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3cfa978e37}} (upgrade dependencies, GHSA-m2qf-hxjv-5gpq)

=== 2023-04-26 ===
* 19:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c6cf151896}} (update dependencies, mainly Flask+Werkzeug 2.3)

=== 2023-02-14 ===
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d498c2ecbd}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes)

=== 2022-09-10 ===
* 18:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8df38d8f43}} (README fix, pulled without webservice restart)
* 16:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ec75eef080}} (gitlab lucaswerkmeister/ → gitlab toolforge-repos/)

=== 2022-07-24 ===
* 16:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|07781279a4}} (fix images with question marks in the title, in a third place)
* 16:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4a08744dc4}} (fix images with question marks in the title, in another place)
* 15:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4793d848d8}} (fix images with question marks in the title)

=== 2022-05-26 ===
* 12:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e878def771}} (two CSS improvements)
* 11:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|43f0101e20}} (Bootstrap 5.1)

=== 2022-05-22 ===
* 20:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a846c4f7f4}} (refactor depicted.css)
* 13:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|15ade38230}} (Diffusion → GitLab)
* 12:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2264a649ae}} (flake8, updated dependencies)
* 09:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4630899d1e}} (tweak message)
* 09:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2bb7a08f06}} (abort on invalid region instead of crashing)

=== 2022-05-20 ===
* 17:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eeb434ac0b}} (add cancel buttons)
* 15:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7abe262b2a}} (remove unused WIP code)
* 13:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f039f87ae0}} (more supported input formats on index page, especially URLs)
* 10:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|010a68cafe}} (image height &lt; viewport height)
* 09:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2fc87dbcf3}} (scroll only image when scaling)

=== 2022-05-17 ===
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|803f7f1f3a}} (info when users can’t edit due to noscript or missing login)

=== 2022-05-15 ===
* 16:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b2fea9010f}} (image scaling support, better width+height+srcset)

=== 2022-05-01 ===
* 17:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ecae67b924}} (fix item ID input handling)
* 15:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|36f85185cb}} (remove QuickStatements support, add strict mode)
* 14:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3fc93364c5}} (add named place on map statements; update cropper.js)

=== 2022-04-30 ===
* 18:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e48c179d47}} (initial support for named place on map, can show and edit regions but not yet add statements)
* 13:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a37bcf7875}} (proper User-Agent)
* 12:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|854eb65b5f}} (use pip-tools; updated some packages, including Flask 2.0.1 → 2.1.2)

=== 2021-11-24 ===
* 13:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61e649c853}} (fix error for some IIIF manifests)

=== 2021-11-03 ===
* 20:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|620e07e107}} (refactoring, no functional change hopefully)

=== 2021-10-28 ===
* 23:21 wm-bot: &lt;lucaswerkmeister&gt; tool is back up but the pre-2021 logs are gone instead of backed up because the container doesn’t have zstd and I didn’t notice in time :(
* 23:19 wm-bot: &lt;lucaswerkmeister&gt; briefly stopping tool (few minutes) to cycle the uwsgi.log

=== 2021-09-25 ===
* 14:46 wm-bot: &lt;lucaswerkmeister&gt; removed old venv-3.5 venv-3.7
* 14:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8797b176c6}} (Python 3.9)

=== 2021-08-02 ===
* 20:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61358b4346}} (one more title and a crash fix)
* 19:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|57b861aaf8}} (useful page title)

=== 2021-07-18 ===
* 19:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eb1f1e04bf}} (update config loading; also updated the venv, including Flask v2)

=== 2021-03-01 ===
* 21:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bf35152db8}} (GitHub actions; adds pytest to venv)

=== 2021-02-28 ===
* 18:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ae6a228597}} (better OAuth error handling)

=== 2021-02-19 ===
* 20:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|73a61f6709}} (avoid mwoauth.identify)

=== 2021-02-16 ===
* 20:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|83700bdd07}} (add skip link)
* 19:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e791f3d956}} (Bootstrap update)

=== 2020-10-17 ===
* 14:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c163184b3a}} (more durable CSRF tokens)

=== 2020-08-08 ===
* 16:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6ec61155ed}} (JS refactoring)

=== 2020-08-07 ===
* 20:22 wm-bot: &lt;lucaswerkmeister&gt; deployed  {{Gerrit|e7fd7f5c82}} (support TIFF etc.)

=== 2020-07-25 ===
* 18:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0dea27dbbd}} (deprecate QuickStatements, minor fixes)

=== 2020-07-22 ===
* 22:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9eb2aa216d}} (no edit region without regions)
* 22:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aa97ea1589}} (Esc for editing regions)
* 21:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6b34c5bb7b}} (editing regions)

=== 2020-06-15 ===
* 21:05 wm-bot: &lt;lucaswerkmeister&gt; renamed default branch from master to main

=== 2020-05-30 ===
* 14:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ff77d74e3f}} (add somevalue depicts statements)

=== 2020-04-25 ===
* 17:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|03e5c7016a}} (fix image.js)
* 17:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5fca9d7d94}} (toolforge.org)
* 16:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|225aea85f1}} (python3.7)

=== 2020-03-28 ===
* 20:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|99ed67d128}} (fix rotated JPEGs some more)
* 14:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|152af31cca}} (fix rotated JPEGs)

=== 2020-03-21 ===
* 14:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|52678de195}} (handle missing/invalid titles and langcode improvement)

=== 2020-02-28 ===
* 21:03 wm-bot: &lt;root&gt; Migrated to 2020 Kubernetes cluster

=== 2020-02-09 ===
* 11:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|97900da63e}} (another language code improvement)
* 06:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5aa0cf4ccb}} (better language code handling)

=== 2019-12-01 ===
* 19:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b72b69d7cb}} (item selector for adding new statements)

=== 2019-11-29 ===
* 23:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f07ce73792}} (fixes for somevalue/novalue and user script)

=== 2019-11-23 ===
* 21:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5eaaa61ffd}} (add new depicteds, some other improvements)
* 14:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|14459bcd4e}} (edit somevalue/novalue)
* 14:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c0b3450603}} (remove legacy API)
* 14:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|448bf42367}} (show depicted somevalue/novalue, and committed no-longer-temporary experiment for OAuth errors)

=== 2019-11-17 ===
* 18:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ef3af6b29f}} (clean up underscore handling; temporary experiment for OAuth errors still in effect)

=== 2019-11-15 ===
* 20:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|57735fe450}} (fix overflow behavior; temporary experiment for OAuth errors still in effect)

=== 2019-11-01 ===
* 11:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|520c618b5a}} (work around [[phab:T222159|T222159]]); temporary experiment still in effect

=== 2019-10-25 ===
* 21:07 wm-bot: &lt;lucaswerkmeister&gt; deployed temporary experiment (clear session on OAuth callback error)
* 15:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7a4982705c}} (improve headings)
* 14:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2d28f55912}} (optimize cropper image loading)

=== 2019-10-24 ===
* 20:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b5dd0fdb31}} (bugfix)

=== 2019-10-23 ===
* 21:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ceb0b0cf8f}} (Structured Data on Commons support, use cropperjs; also tried a general pip upgrade but that broke OAuth login so downgraded to previous versions, except pip itself)

=== 2019-05-20 ===
* 09:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dfaa0c4093}} (switch to Python 3.5), including venv rebuild
* 09:23 wm-bot: &lt;lucaswerkmeister&gt; stopping webserver for Python 3.5 upgrade


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>ns1kfxtjxez7holyimyhkbt80hoe18y</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebot3/SAL</title>
    <ns>498</ns>
    <id>446211</id>
    <revision>
      <id>2478982</id>
      <parentid>2478930</parentid>
      <timestamp>2026-11-07T12:34:36Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819831 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478982</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="14219" sha1="nfbm6a74z80uus7pj1764i0eetfetvv" xml:space="preserve">=== 2026-11-07 ===
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819831 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42301732341 (https://github.com/cluebotng/component-configs/commits/2359fb30db198ebb0e5c48ceade9726f34292e61)
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564857 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723424 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248627903 (https://github.com/cluebotng/component-configs/commits/a0ce1e9c453162937a059133fd92b52fde5d6083)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233912183 (https://github.com/cluebotng/component-configs/commits/5de5c8fa4679db8df6d651ae192b27cc21927046)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833480 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249044 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188028 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297593 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300400 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208889 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616290 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264139 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529465 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610868 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626121 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125832150 (https://github.com/cluebotng/component-configs/commits/4fe5c34ec5605757ac5b0f1c977bd5fe785ea4df)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901676263 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757798 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40899035740 (https://github.com/cluebotng/component-configs/commits/46b9114bc14193bf52f6200273cd59584dbe78e5)
* 15:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898308339 (https://github.com/cluebotng/component-configs/commits/098b7b2c89711b56ef558c0b46e37906974cac41)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266243 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788418 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508590936 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40492675207 (https://github.com/cluebotng/component-configs/commits/677efd2e385dcbac779a72a750798b6b7fa31221)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518152 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375517 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527024 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)

=== 2026-10-23 ===
* 16:26 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-10-09 ===
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/37949440447 (https://github.com/cluebotng/component-configs/commits/902367266d74fd0d5bf2343e1502e8e1eefc330c)

=== 2026-10-01 ===
* 15:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881820713 (https://github.com/cluebotng/component-configs/commits/860f13c0de005840e3e832d5dc6d17d9a4863409)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881408868 (https://github.com/cluebotng/component-configs/commits/2eac12b733c0a202e1363dde61dd15496bc3d385)

=== 2026-09-29 ===
* 11:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36560726793 (https://github.com/cluebotng/component-configs/commits/8638ec82624573a20aa5fbc7d9b44374a34b35e4)

=== 2026-09-14 ===
* 11:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/34839795990 (https://github.com/cluebotng/component-configs/commits/a4485cc4a40f2edd0ef8fec82c97814550c9585c)

=== 2026-09-04 ===
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33872950476 (https://github.com/cluebotng/component-configs/commits/5308018031c449d9ac37a3c10524e02cee50ec8b)

=== 2026-09-03 ===
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33738502759 (https://github.com/cluebotng/component-configs/commits/a57be2aa8221abd40bf701a3d617fddcec5fd47b)

=== 2026-09-01 ===
* 16:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531533466 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-31 ===
* 17:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33421096964 (https://github.com/cluebotng/component-configs/commits/50d17b37689aacfab96d6a5683d74be969ee0bc3)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415531754 (https://github.com/cluebotng/component-configs/commits/fa3e6f3691a39e901537a1debc031329ff5c3eeb)
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415131548 (https://github.com/cluebotng/component-configs/commits/cd0c47ddd7ab13987792b87be0155ebff12a87bb)
* 16:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33411677163 (https://github.com/cluebotng/component-configs/commits/4ce131b4635a642a351fd11a9bfe616b5cf675a4)
* 15:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33410122686 (https://github.com/cluebotng/component-configs/commits/f04e19b5856fdc410758a31113106c5d61074e04)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412758 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506454 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770905 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126281 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788675 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27268776225 (https://github.com/cluebotng/component-configs/commits/83d2cb83b77ce82739917e8a6f69d2fa261a35f0)

=== 2026-05-31 ===
* 17:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26719587601 (https://github.com/cluebotng/component-configs/commits/a9b897ef152e0487191e34674f50ee5ffaba37d5)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442392 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247576870 (https://github.com/cluebotng/component-configs/commits/09b7afbf5d13900afa4cbcc31de73b324f45b3a9)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640342 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381611326 (https://github.com/cluebotng/component-configs/commits/c56f3042ae5b94ccb7092edbffa3debdc80428c5)
* 14:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381564784 (https://github.com/cluebotng/component-configs/commits/0f5ac0701dbabd4d5dd8f7ab1f2ba160b5be4f07)

=== 2026-02-15 ===
* 13:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22036393808 (https://github.com/cluebotng/component-configs/commits/edb625f4e3a338fafceb15b2d9990b27e45658cf)

=== 2025-11-12 ===
* 07:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289349325 (https://github.com/cluebotng/component-configs/commits/ac67bbd04ad7d30de28f9a7daecf5a7d136f62ef)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19288653346 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642903 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270254406 (https://github.com/cluebotng/component-configs/commits/5dfa0eff0c51e0d878863a5ac689e59f79d59876)

=== 2025-08-25 ===
* 11:24 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.1

=== 2025-08-24 ===
* 16:33 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.0

=== 2025-08-18 ===
* 15:21 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.23
* 15:09 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.22
* 14:59 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.21
* 13:32 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.18
* 13:11 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.17
* 13:03 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.16
* 12:55 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.15
* 12:39 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.14

=== 2025-08-07 ===
* 17:07 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.5
* 15:38 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.4

=== 2020-07-02 ===
* 09:16 RichSmith: Restarted CB3 after reports it was down


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>nfbm6a74z80uus7pj1764i0eetfetvv</sha1>
    </revision>
    <revision>
      <id>2478988</id>
      <parentid>2478982</parentid>
      <timestamp>2026-11-07T12:37:31Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017594 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)</comment>
      <origin>2478988</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="14425" sha1="9ah3gb9uat6ha8jyxnudcp0kjf605qu" xml:space="preserve">=== 2026-11-07 ===
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017594 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819831 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42301732341 (https://github.com/cluebotng/component-configs/commits/2359fb30db198ebb0e5c48ceade9726f34292e61)
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564857 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723424 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248627903 (https://github.com/cluebotng/component-configs/commits/a0ce1e9c453162937a059133fd92b52fde5d6083)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233912183 (https://github.com/cluebotng/component-configs/commits/5de5c8fa4679db8df6d651ae192b27cc21927046)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833480 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249044 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188028 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297593 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300400 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208889 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616290 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264139 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529465 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610868 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626121 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125832150 (https://github.com/cluebotng/component-configs/commits/4fe5c34ec5605757ac5b0f1c977bd5fe785ea4df)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901676263 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757798 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40899035740 (https://github.com/cluebotng/component-configs/commits/46b9114bc14193bf52f6200273cd59584dbe78e5)
* 15:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898308339 (https://github.com/cluebotng/component-configs/commits/098b7b2c89711b56ef558c0b46e37906974cac41)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266243 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788418 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508590936 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40492675207 (https://github.com/cluebotng/component-configs/commits/677efd2e385dcbac779a72a750798b6b7fa31221)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518152 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375517 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527024 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)

=== 2026-10-23 ===
* 16:26 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-10-09 ===
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/37949440447 (https://github.com/cluebotng/component-configs/commits/902367266d74fd0d5bf2343e1502e8e1eefc330c)

=== 2026-10-01 ===
* 15:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881820713 (https://github.com/cluebotng/component-configs/commits/860f13c0de005840e3e832d5dc6d17d9a4863409)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881408868 (https://github.com/cluebotng/component-configs/commits/2eac12b733c0a202e1363dde61dd15496bc3d385)

=== 2026-09-29 ===
* 11:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36560726793 (https://github.com/cluebotng/component-configs/commits/8638ec82624573a20aa5fbc7d9b44374a34b35e4)

=== 2026-09-14 ===
* 11:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/34839795990 (https://github.com/cluebotng/component-configs/commits/a4485cc4a40f2edd0ef8fec82c97814550c9585c)

=== 2026-09-04 ===
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33872950476 (https://github.com/cluebotng/component-configs/commits/5308018031c449d9ac37a3c10524e02cee50ec8b)

=== 2026-09-03 ===
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33738502759 (https://github.com/cluebotng/component-configs/commits/a57be2aa8221abd40bf701a3d617fddcec5fd47b)

=== 2026-09-01 ===
* 16:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531533466 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-31 ===
* 17:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33421096964 (https://github.com/cluebotng/component-configs/commits/50d17b37689aacfab96d6a5683d74be969ee0bc3)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415531754 (https://github.com/cluebotng/component-configs/commits/fa3e6f3691a39e901537a1debc031329ff5c3eeb)
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415131548 (https://github.com/cluebotng/component-configs/commits/cd0c47ddd7ab13987792b87be0155ebff12a87bb)
* 16:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33411677163 (https://github.com/cluebotng/component-configs/commits/4ce131b4635a642a351fd11a9bfe616b5cf675a4)
* 15:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33410122686 (https://github.com/cluebotng/component-configs/commits/f04e19b5856fdc410758a31113106c5d61074e04)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412758 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506454 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770905 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126281 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788675 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27268776225 (https://github.com/cluebotng/component-configs/commits/83d2cb83b77ce82739917e8a6f69d2fa261a35f0)

=== 2026-05-31 ===
* 17:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26719587601 (https://github.com/cluebotng/component-configs/commits/a9b897ef152e0487191e34674f50ee5ffaba37d5)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442392 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247576870 (https://github.com/cluebotng/component-configs/commits/09b7afbf5d13900afa4cbcc31de73b324f45b3a9)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640342 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381611326 (https://github.com/cluebotng/component-configs/commits/c56f3042ae5b94ccb7092edbffa3debdc80428c5)
* 14:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381564784 (https://github.com/cluebotng/component-configs/commits/0f5ac0701dbabd4d5dd8f7ab1f2ba160b5be4f07)

=== 2026-02-15 ===
* 13:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22036393808 (https://github.com/cluebotng/component-configs/commits/edb625f4e3a338fafceb15b2d9990b27e45658cf)

=== 2025-11-12 ===
* 07:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289349325 (https://github.com/cluebotng/component-configs/commits/ac67bbd04ad7d30de28f9a7daecf5a7d136f62ef)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19288653346 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642903 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270254406 (https://github.com/cluebotng/component-configs/commits/5dfa0eff0c51e0d878863a5ac689e59f79d59876)

=== 2025-08-25 ===
* 11:24 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.1

=== 2025-08-24 ===
* 16:33 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.0

=== 2025-08-18 ===
* 15:21 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.23
* 15:09 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.22
* 14:59 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.21
* 13:32 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.18
* 13:11 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.17
* 13:03 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.16
* 12:55 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.15
* 12:39 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.14

=== 2025-08-07 ===
* 17:07 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.5
* 15:38 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.4

=== 2020-07-02 ===
* 09:16 RichSmith: Restarted CB3 after reports it was down


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>9ah3gb9uat6ha8jyxnudcp0kjf605qu</sha1>
    </revision>
    <revision>
      <id>2478998</id>
      <parentid>2478988</parentid>
      <timestamp>2026-11-07T12:43:53Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373444118 (https://github.com/cluebotng/component-configs/commits/4b7bb61eba3f6cf3781522588941262f6b984ed5)</comment>
      <origin>2478998</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="14631" sha1="e8ew65lb6yjtrnvqhv80vx3vduj088a" xml:space="preserve">=== 2026-11-07 ===
* 12:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373444118 (https://github.com/cluebotng/component-configs/commits/4b7bb61eba3f6cf3781522588941262f6b984ed5)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017594 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819831 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42301732341 (https://github.com/cluebotng/component-configs/commits/2359fb30db198ebb0e5c48ceade9726f34292e61)
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564857 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723424 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248627903 (https://github.com/cluebotng/component-configs/commits/a0ce1e9c453162937a059133fd92b52fde5d6083)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233912183 (https://github.com/cluebotng/component-configs/commits/5de5c8fa4679db8df6d651ae192b27cc21927046)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833480 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249044 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188028 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297593 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300400 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208889 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616290 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264139 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529465 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610868 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626121 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125832150 (https://github.com/cluebotng/component-configs/commits/4fe5c34ec5605757ac5b0f1c977bd5fe785ea4df)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901676263 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757798 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40899035740 (https://github.com/cluebotng/component-configs/commits/46b9114bc14193bf52f6200273cd59584dbe78e5)
* 15:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898308339 (https://github.com/cluebotng/component-configs/commits/098b7b2c89711b56ef558c0b46e37906974cac41)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266243 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788418 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508590936 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40492675207 (https://github.com/cluebotng/component-configs/commits/677efd2e385dcbac779a72a750798b6b7fa31221)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518152 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375517 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527024 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)

=== 2026-10-23 ===
* 16:26 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-10-09 ===
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/37949440447 (https://github.com/cluebotng/component-configs/commits/902367266d74fd0d5bf2343e1502e8e1eefc330c)

=== 2026-10-01 ===
* 15:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881820713 (https://github.com/cluebotng/component-configs/commits/860f13c0de005840e3e832d5dc6d17d9a4863409)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881408868 (https://github.com/cluebotng/component-configs/commits/2eac12b733c0a202e1363dde61dd15496bc3d385)

=== 2026-09-29 ===
* 11:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36560726793 (https://github.com/cluebotng/component-configs/commits/8638ec82624573a20aa5fbc7d9b44374a34b35e4)

=== 2026-09-14 ===
* 11:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/34839795990 (https://github.com/cluebotng/component-configs/commits/a4485cc4a40f2edd0ef8fec82c97814550c9585c)

=== 2026-09-04 ===
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33872950476 (https://github.com/cluebotng/component-configs/commits/5308018031c449d9ac37a3c10524e02cee50ec8b)

=== 2026-09-03 ===
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33738502759 (https://github.com/cluebotng/component-configs/commits/a57be2aa8221abd40bf701a3d617fddcec5fd47b)

=== 2026-09-01 ===
* 16:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531533466 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-31 ===
* 17:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33421096964 (https://github.com/cluebotng/component-configs/commits/50d17b37689aacfab96d6a5683d74be969ee0bc3)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415531754 (https://github.com/cluebotng/component-configs/commits/fa3e6f3691a39e901537a1debc031329ff5c3eeb)
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415131548 (https://github.com/cluebotng/component-configs/commits/cd0c47ddd7ab13987792b87be0155ebff12a87bb)
* 16:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33411677163 (https://github.com/cluebotng/component-configs/commits/4ce131b4635a642a351fd11a9bfe616b5cf675a4)
* 15:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33410122686 (https://github.com/cluebotng/component-configs/commits/f04e19b5856fdc410758a31113106c5d61074e04)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412758 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506454 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770905 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126281 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788675 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27268776225 (https://github.com/cluebotng/component-configs/commits/83d2cb83b77ce82739917e8a6f69d2fa261a35f0)

=== 2026-05-31 ===
* 17:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26719587601 (https://github.com/cluebotng/component-configs/commits/a9b897ef152e0487191e34674f50ee5ffaba37d5)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442392 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247576870 (https://github.com/cluebotng/component-configs/commits/09b7afbf5d13900afa4cbcc31de73b324f45b3a9)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640342 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381611326 (https://github.com/cluebotng/component-configs/commits/c56f3042ae5b94ccb7092edbffa3debdc80428c5)
* 14:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381564784 (https://github.com/cluebotng/component-configs/commits/0f5ac0701dbabd4d5dd8f7ab1f2ba160b5be4f07)

=== 2026-02-15 ===
* 13:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22036393808 (https://github.com/cluebotng/component-configs/commits/edb625f4e3a338fafceb15b2d9990b27e45658cf)

=== 2025-11-12 ===
* 07:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289349325 (https://github.com/cluebotng/component-configs/commits/ac67bbd04ad7d30de28f9a7daecf5a7d136f62ef)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19288653346 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642903 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270254406 (https://github.com/cluebotng/component-configs/commits/5dfa0eff0c51e0d878863a5ac689e59f79d59876)

=== 2025-08-25 ===
* 11:24 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.1

=== 2025-08-24 ===
* 16:33 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.0

=== 2025-08-18 ===
* 15:21 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.23
* 15:09 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.22
* 14:59 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.21
* 13:32 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.18
* 13:11 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.17
* 13:03 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.16
* 12:55 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.15
* 12:39 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.14

=== 2025-08-07 ===
* 17:07 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.5
* 15:38 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.4

=== 2020-07-02 ===
* 09:16 RichSmith: Restarted CB3 after reports it was down


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>e8ew65lb6yjtrnvqhv80vx3vduj088a</sha1>
    </revision>
    <revision>
      <id>2479003</id>
      <parentid>2478998</parentid>
      <timestamp>2026-11-07T12:47:04Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373456031 (https://github.com/cluebotng/component-configs/commits/9b75826d2873a10720e1149d8b0e5aecd8df070f)</comment>
      <origin>2479003</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="14837" sha1="20zge0q39xab2nxw6yct2cxla2zqv5o" xml:space="preserve">=== 2026-11-07 ===
* 12:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373456031 (https://github.com/cluebotng/component-configs/commits/9b75826d2873a10720e1149d8b0e5aecd8df070f)
* 12:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373444118 (https://github.com/cluebotng/component-configs/commits/4b7bb61eba3f6cf3781522588941262f6b984ed5)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017594 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819831 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42301732341 (https://github.com/cluebotng/component-configs/commits/2359fb30db198ebb0e5c48ceade9726f34292e61)
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564857 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723424 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248627903 (https://github.com/cluebotng/component-configs/commits/a0ce1e9c453162937a059133fd92b52fde5d6083)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233912183 (https://github.com/cluebotng/component-configs/commits/5de5c8fa4679db8df6d651ae192b27cc21927046)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833480 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249044 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188028 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297593 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300400 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208889 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616290 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264139 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529465 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610868 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626121 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125832150 (https://github.com/cluebotng/component-configs/commits/4fe5c34ec5605757ac5b0f1c977bd5fe785ea4df)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901676263 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757798 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40899035740 (https://github.com/cluebotng/component-configs/commits/46b9114bc14193bf52f6200273cd59584dbe78e5)
* 15:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898308339 (https://github.com/cluebotng/component-configs/commits/098b7b2c89711b56ef558c0b46e37906974cac41)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266243 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788418 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508590936 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40492675207 (https://github.com/cluebotng/component-configs/commits/677efd2e385dcbac779a72a750798b6b7fa31221)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518152 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375517 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527024 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)

=== 2026-10-23 ===
* 16:26 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-10-09 ===
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/37949440447 (https://github.com/cluebotng/component-configs/commits/902367266d74fd0d5bf2343e1502e8e1eefc330c)

=== 2026-10-01 ===
* 15:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881820713 (https://github.com/cluebotng/component-configs/commits/860f13c0de005840e3e832d5dc6d17d9a4863409)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881408868 (https://github.com/cluebotng/component-configs/commits/2eac12b733c0a202e1363dde61dd15496bc3d385)

=== 2026-09-29 ===
* 11:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36560726793 (https://github.com/cluebotng/component-configs/commits/8638ec82624573a20aa5fbc7d9b44374a34b35e4)

=== 2026-09-14 ===
* 11:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/34839795990 (https://github.com/cluebotng/component-configs/commits/a4485cc4a40f2edd0ef8fec82c97814550c9585c)

=== 2026-09-04 ===
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33872950476 (https://github.com/cluebotng/component-configs/commits/5308018031c449d9ac37a3c10524e02cee50ec8b)

=== 2026-09-03 ===
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33738502759 (https://github.com/cluebotng/component-configs/commits/a57be2aa8221abd40bf701a3d617fddcec5fd47b)

=== 2026-09-01 ===
* 16:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531533466 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-31 ===
* 17:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33421096964 (https://github.com/cluebotng/component-configs/commits/50d17b37689aacfab96d6a5683d74be969ee0bc3)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415531754 (https://github.com/cluebotng/component-configs/commits/fa3e6f3691a39e901537a1debc031329ff5c3eeb)
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415131548 (https://github.com/cluebotng/component-configs/commits/cd0c47ddd7ab13987792b87be0155ebff12a87bb)
* 16:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33411677163 (https://github.com/cluebotng/component-configs/commits/4ce131b4635a642a351fd11a9bfe616b5cf675a4)
* 15:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33410122686 (https://github.com/cluebotng/component-configs/commits/f04e19b5856fdc410758a31113106c5d61074e04)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412758 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506454 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770905 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126281 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788675 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27268776225 (https://github.com/cluebotng/component-configs/commits/83d2cb83b77ce82739917e8a6f69d2fa261a35f0)

=== 2026-05-31 ===
* 17:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26719587601 (https://github.com/cluebotng/component-configs/commits/a9b897ef152e0487191e34674f50ee5ffaba37d5)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442392 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247576870 (https://github.com/cluebotng/component-configs/commits/09b7afbf5d13900afa4cbcc31de73b324f45b3a9)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640342 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381611326 (https://github.com/cluebotng/component-configs/commits/c56f3042ae5b94ccb7092edbffa3debdc80428c5)
* 14:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381564784 (https://github.com/cluebotng/component-configs/commits/0f5ac0701dbabd4d5dd8f7ab1f2ba160b5be4f07)

=== 2026-02-15 ===
* 13:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22036393808 (https://github.com/cluebotng/component-configs/commits/edb625f4e3a338fafceb15b2d9990b27e45658cf)

=== 2025-11-12 ===
* 07:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289349325 (https://github.com/cluebotng/component-configs/commits/ac67bbd04ad7d30de28f9a7daecf5a7d136f62ef)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19288653346 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642903 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270254406 (https://github.com/cluebotng/component-configs/commits/5dfa0eff0c51e0d878863a5ac689e59f79d59876)

=== 2025-08-25 ===
* 11:24 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.1

=== 2025-08-24 ===
* 16:33 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.0

=== 2025-08-18 ===
* 15:21 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.23
* 15:09 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.22
* 14:59 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.21
* 13:32 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.18
* 13:11 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.17
* 13:03 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.16
* 12:55 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.15
* 12:39 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.14

=== 2025-08-07 ===
* 17:07 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.5
* 15:38 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.4

=== 2020-07-02 ===
* 09:16 RichSmith: Restarted CB3 after reports it was down


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>20zge0q39xab2nxw6yct2cxla2zqv5o</sha1>
    </revision>
    <revision>
      <id>2479016</id>
      <parentid>2479003</parentid>
      <timestamp>2026-11-08T11:26:32Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373456031 (https://github.com/cluebotng/component-configs/commits/9b75826d2873a10720e1149d8b0e5aecd8df070f)</comment>
      <origin>2479016</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="15063" sha1="h9pttoxoetztdwi8txel13u3ms5knda" xml:space="preserve">=== 2026-11-08 ===
* 11:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373456031 (https://github.com/cluebotng/component-configs/commits/9b75826d2873a10720e1149d8b0e5aecd8df070f)

=== 2026-11-07 ===
* 12:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373456031 (https://github.com/cluebotng/component-configs/commits/9b75826d2873a10720e1149d8b0e5aecd8df070f)
* 12:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373444118 (https://github.com/cluebotng/component-configs/commits/4b7bb61eba3f6cf3781522588941262f6b984ed5)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017594 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819831 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42301732341 (https://github.com/cluebotng/component-configs/commits/2359fb30db198ebb0e5c48ceade9726f34292e61)
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564857 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723424 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248627903 (https://github.com/cluebotng/component-configs/commits/a0ce1e9c453162937a059133fd92b52fde5d6083)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233912183 (https://github.com/cluebotng/component-configs/commits/5de5c8fa4679db8df6d651ae192b27cc21927046)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833480 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249044 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188028 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297593 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300400 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208889 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616290 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264139 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529465 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610868 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626121 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125832150 (https://github.com/cluebotng/component-configs/commits/4fe5c34ec5605757ac5b0f1c977bd5fe785ea4df)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901676263 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757798 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40899035740 (https://github.com/cluebotng/component-configs/commits/46b9114bc14193bf52f6200273cd59584dbe78e5)
* 15:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898308339 (https://github.com/cluebotng/component-configs/commits/098b7b2c89711b56ef558c0b46e37906974cac41)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266243 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788418 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508590936 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40492675207 (https://github.com/cluebotng/component-configs/commits/677efd2e385dcbac779a72a750798b6b7fa31221)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518152 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375517 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527024 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)

=== 2026-10-23 ===
* 16:26 Damianz: re-enabled under components, nfs based hack cleaned up
* 10:47 Damianz: running under manual job using pre-built 8.4 image until buildpack support returns (repo/composer install in home dir)

=== 2026-10-09 ===
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/37949440447 (https://github.com/cluebotng/component-configs/commits/902367266d74fd0d5bf2343e1502e8e1eefc330c)

=== 2026-10-01 ===
* 15:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881820713 (https://github.com/cluebotng/component-configs/commits/860f13c0de005840e3e832d5dc6d17d9a4863409)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36881408868 (https://github.com/cluebotng/component-configs/commits/2eac12b733c0a202e1363dde61dd15496bc3d385)

=== 2026-09-29 ===
* 11:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/36560726793 (https://github.com/cluebotng/component-configs/commits/8638ec82624573a20aa5fbc7d9b44374a34b35e4)

=== 2026-09-14 ===
* 11:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/34839795990 (https://github.com/cluebotng/component-configs/commits/a4485cc4a40f2edd0ef8fec82c97814550c9585c)

=== 2026-09-04 ===
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33872950476 (https://github.com/cluebotng/component-configs/commits/5308018031c449d9ac37a3c10524e02cee50ec8b)

=== 2026-09-03 ===
* 09:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33738502759 (https://github.com/cluebotng/component-configs/commits/a57be2aa8221abd40bf701a3d617fddcec5fd47b)

=== 2026-09-01 ===
* 16:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33531533466 (https://github.com/cluebotng/component-configs/commits/348098934d025cf69ee0caa7262b0dfd90ad0b14)

=== 2026-08-31 ===
* 17:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33421096964 (https://github.com/cluebotng/component-configs/commits/50d17b37689aacfab96d6a5683d74be969ee0bc3)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415531754 (https://github.com/cluebotng/component-configs/commits/fa3e6f3691a39e901537a1debc031329ff5c3eeb)
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33415131548 (https://github.com/cluebotng/component-configs/commits/cd0c47ddd7ab13987792b87be0155ebff12a87bb)
* 16:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33411677163 (https://github.com/cluebotng/component-configs/commits/4ce131b4635a642a351fd11a9bfe616b5cf675a4)
* 15:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33410122686 (https://github.com/cluebotng/component-configs/commits/f04e19b5856fdc410758a31113106c5d61074e04)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412758 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506454 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770905 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126281 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788675 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27268776225 (https://github.com/cluebotng/component-configs/commits/83d2cb83b77ce82739917e8a6f69d2fa261a35f0)

=== 2026-05-31 ===
* 17:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26719587601 (https://github.com/cluebotng/component-configs/commits/a9b897ef152e0487191e34674f50ee5ffaba37d5)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442392 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247576870 (https://github.com/cluebotng/component-configs/commits/09b7afbf5d13900afa4cbcc31de73b324f45b3a9)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640342 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381611326 (https://github.com/cluebotng/component-configs/commits/c56f3042ae5b94ccb7092edbffa3debdc80428c5)
* 14:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381564784 (https://github.com/cluebotng/component-configs/commits/0f5ac0701dbabd4d5dd8f7ab1f2ba160b5be4f07)

=== 2026-02-15 ===
* 13:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22036393808 (https://github.com/cluebotng/component-configs/commits/edb625f4e3a338fafceb15b2d9990b27e45658cf)

=== 2025-11-12 ===
* 07:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289349325 (https://github.com/cluebotng/component-configs/commits/ac67bbd04ad7d30de28f9a7daecf5a7d136f62ef)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19288653346 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642903 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270254406 (https://github.com/cluebotng/component-configs/commits/5dfa0eff0c51e0d878863a5ac689e59f79d59876)

=== 2025-08-25 ===
* 11:24 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.1

=== 2025-08-24 ===
* 16:33 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.2.0

=== 2025-08-18 ===
* 15:21 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.23
* 15:09 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.22
* 14:59 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.21
* 13:32 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.18
* 13:11 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.17
* 13:03 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.16
* 12:55 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.15
* 12:39 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.0.14

=== 2025-08-07 ===
* 17:07 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.5
* 15:38 wmbot~damian@tools-bastion-13: bot deployed @ refs/tags/v1.1.4

=== 2020-07-02 ===
* 09:16 RichSmith: Restarted CB3 after reports it was down


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>h9pttoxoetztdwi8txel13u3ms5knda</sha1>
    </revision>
  </page>
  <page>
    <title>Map of database maintenance</title>
    <ns>0</ns>
    <id>449160</id>
    <revision>
      <id>2479012</id>
      <parentid>2478947</parentid>
      <timestamp>2026-11-08T00:00:33Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2479012</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="672" sha1="5x99fteiu0n1lsnhetvnc4gr3ws7y00" xml:space="preserve">{{/Header}}
== Today (2026-11-08) ==
== Yesterday (2026-11-07) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| s2 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
| s3 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
| s5 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
| s7 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
| s8 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| s2 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
| s7 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
| s8 || [[phab:T441633|Login (T441633)]] (ladsgroup)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>5x99fteiu0n1lsnhetvnc4gr3ws7y00</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-review/SAL</title>
    <ns>498</ns>
    <id>452890</id>
    <revision>
      <id>2478974</id>
      <parentid>2478959</parentid>
      <timestamp>2026-11-07T12:31:50Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819771 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478974</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="58855" sha1="eayamq1o81ooqliv4j6sah9pir7g1zi" xml:space="preserve">=== 2026-11-07 ===
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819771 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251051 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 16:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695395 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249485663 (https://github.com/cluebotng/component-configs/commits/8c7f8b7d54d8637bd5f421ad2f493b96a7008504)
* 16:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248508387 (https://github.com/cluebotng/component-configs/commits/3acae96966e16f4f9f91683ad36f5e56d329753a)
* 14:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234031860 (https://github.com/cluebotng/component-configs/commits/15c352e208f001eee46b8462013282f7e93d9872)
* 14:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833470 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248930 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188041 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 13:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42223625514 (https://github.com/cluebotng/component-configs/commits/9b07be56cd997c4d5787c311dd3e89941f8e528a)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42222102051 (https://github.com/cluebotng/component-configs/commits/5a9290a457af2e5700d955bc65fbd8d77cad712c)
* 12:21 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485332 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 07:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42181945109 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42177270550 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297558 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42166677986 (https://github.com/cluebotng/component-configs/commits/ceeb81f4d5e5d7b6a4947b95df020093bdea2903)
* 04:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42163050761 (https://github.com/cluebotng/component-configs/commits/82355b13bf40529ca3af84e6749306f7cad42b92)
* 03:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618011 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 15:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42052229859 (https://github.com/cluebotng/component-configs/commits/4f5b065a67c9c1036d100bb6ae2816f193b11b90)
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42049573846 (https://github.com/cluebotng/component-configs/commits/120177aabd7236410755f7187b351195daceb172)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978491372 (https://github.com/cluebotng/component-configs/commits/c13b42c494b46b9b47c56144b4bfef621ce4246b)
* 05:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41976470115 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300319 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 04:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41970294033 (https://github.com/cluebotng/component-configs/commits/249d7f9d3e64c31431d63973b2e535b32af4e788)
* 03:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41961409152 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 23:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942779 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41871475015 (https://github.com/cluebotng/component-configs/commits/2013e4c5bd958fdebe0a0c4511369f5a2457b368)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41777591788 (https://github.com/cluebotng/component-configs/commits/e816c7dcfa9f55dfe3171cd83dcb396d239b99eb)
* 04:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41773053027 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208843 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274459 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119500 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41694203768 (https://github.com/cluebotng/component-configs/commits/50df9beed114de84489c4353abb75acc5cc118bd)
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693690557 (https://github.com/cluebotng/component-configs/commits/2350c7e21a6306b68d1249b6635e7820e90f8d65)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616142 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 06:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264177 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147205 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5)
* 05:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41583533049 (https://github.com/cluebotng/component-configs/commits/417cde41c250a5f0b6f21e72ce00a343ad270295)
* 02:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216266 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434964 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 16:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41220829549 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 14:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074752 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347132 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603064 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529432 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726841 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 09:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186724200 (https://github.com/cluebotng/component-configs/commits/8474642453be7b488acccbcd8e7bfe5261236e6c)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186122222 (https://github.com/cluebotng/component-configs/commits/87121d4752d813dd9d7319b0d336491ee1ee0cb5)
* 08:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002301 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595340 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610852 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626232 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41135430871 (https://github.com/cluebotng/component-configs/commits/118e964d982be74b124949a21f67c966a2b2bea0)
* 21:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558348 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761815 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)
* 19:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41112155068 (https://github.com/cluebotng/component-configs/commits/2e97d5ca05d7bdca41d16a18c30c1820a9b1eb1c)

=== 2026-10-29 ===
* 15:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40903315472 (https://github.com/cluebotng/component-configs/commits/48d12f535986ffcf04bdc857a204667a10e4579d)
* 15:38 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696710 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902023817 (https://github.com/cluebotng/component-configs/commits/d8c689f56957540b9a4fd6c51ce60701a2b483c9)
* 15:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900617605 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898273176 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266172 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)
* 14:48 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40894527337 (https://github.com/cluebotng/component-configs/commits/b6e74433624164c9f93d76d8a23016bfa61d9481)

=== 2026-10-28 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507977 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788346 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987481 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc)
* 05:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879060 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)
* 02:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021329 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf)

=== 2026-10-27 ===
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40547448775 (https://github.com/cluebotng/component-configs/commits/d8e5c2b81841263635a002cdb005aaa7328042b8)
* 16:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112186 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 16:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846798 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 15:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588887 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106)
* 14:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783532 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 14:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784076 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591099 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231000 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21)
* 10:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40491080407 (https://github.com/cluebotng/component-configs/commits/d72ea811ba6c14ea73c3b76fe67040cdac4fa091)
* 09:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261191 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249549 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601965 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed)
* 06:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540119 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457745340 (https://github.com/cluebotng/component-configs/commits/c09fe8cd0f8bc989e3798ba689cbffc5c5cb362a)
* 05:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798133 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721244 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 05:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518090 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765768 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375770 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526960 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183898 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)
* 03:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895976 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40441575561 (https://github.com/cluebotng/component-configs/commits/457585647825bf9845a8c2a5e6ab8dba5aca3763)

=== 2026-08-29 ===
* 12:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33252834244 (https://github.com/cluebotng/component-configs/commits/4cbc4575b18a5c1d868b8284f76bb7ca43fe85d4)
* 10:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33248315658 (https://github.com/cluebotng/component-configs/commits/deff83b4f72b157f24d761127affb1362b599cff)

=== 2026-07-15 ===
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29435050584 (https://github.com/cluebotng/component-configs/commits/912a8ea89e62cddf63a20263572b5c94900ef06a)
* 15:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-02 ===
* 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 20:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28544657044 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861)
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506430 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337269 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 18:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28297170540 (https://github.com/cluebotng/component-configs/commits/12a68f8f80991f77c948817dd50798649df0bfac)
* 17:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28296058232 (https://github.com/cluebotng/component-configs/commits/e1ce8770ee82bd4202c7f54215f5b514b693352d)
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28294935998 (https://github.com/cluebotng/component-configs/commits/dfafa3368a83b4ef570763c77a8e2131ae43e790)
* 16:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28294935998 (https://github.com/cluebotng/component-configs/commits/dfafa3368a83b4ef570763c77a8e2131ae43e790)
* 14:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28292262937 (https://github.com/cluebotng/component-configs/commits/d0d81399db60e997e0f3c84c4bfcd671b88d5e51)

=== 2026-06-24 ===
* 17:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117368498 (https://github.com/cluebotng/component-configs/commits/2a2ca3b7a9ed1a932fd98d3ae6694fc8b386db5d)

=== 2026-06-16 ===
* 17:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849959 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737527 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634025367 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770870 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27562889711 (https://github.com/cluebotng/component-configs/commits/b9c48365007a90ed1c240655306615542751ea6f)
* 17:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27562697218 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2)
* 14:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27554179713 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-14 ===
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27495607670 (https://github.com/cluebotng/component-configs/commits/63eaa8007e638829714e8661427aa311d662b341)

=== 2026-06-13 ===
* 16:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27472937497 (https://github.com/cluebotng/component-configs/commits/38317c72706d30dee2f423e689b46826fa0cbb9a)
* 13:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463970 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)
* 11:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27464936848 (https://github.com/cluebotng/component-configs/commits/d19537391f153a63eba67672cf3aecb76bff362e)

=== 2026-06-11 ===
* 13:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27350281127 (https://github.com/cluebotng/component-configs/commits/41e666d399f22121043fa8132a3e8bf6368e5181)

=== 2026-06-10 ===
* 15:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285125877 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277789050 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276698071 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276511398 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275411814 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339348 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27225356873 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156886 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601200 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)
* 14:04 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27019408986 (https://github.com/cluebotng/component-configs/commits/4de86ac4b524f394ec49dc167d0c75457e981af1)
* 12:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27016160066 (https://github.com/cluebotng/component-configs/commits/3c617707fe44f00c5ac81aa315bfc989a7bd4d00)
* 11:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27012814543 (https://github.com/cluebotng/component-configs/commits/48e7143fc26338766785004996ec66dea5876b0e)
* 11:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27011683531 (https://github.com/cluebotng/component-configs/commits/7b3ba4bae58717e47510c281fdb60f00be6f7c92)

=== 2026-06-01 ===
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978339 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757971428 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864796 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862181 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719681637 (https://github.com/cluebotng/component-configs/commits/02759bc368e1f9901ca399942295a807038321cb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575507 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d)

=== 2026-05-30 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26691257350 (https://github.com/cluebotng/component-configs/commits/1daccd14ff5fe952e32175ee2cf249f2312d99ae)

=== 2026-05-29 ===
* 00:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639469 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20)

=== 2026-05-28 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792853 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8)

=== 2026-05-21 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627549 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056)

=== 2026-05-19 ===
* 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076396 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c)

=== 2026-05-17 ===
* 08:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25727821525 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842068 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562210 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25723480598 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280529 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-06 ===
* 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-02 ===
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392660 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)

=== 2026-04-20 ===
* 23:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1)

=== 2026-04-17 ===
* 00:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422836848 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-11 ===
* 10:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2)
* 10:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1)

=== 2026-04-10 ===
* 23:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268430819 (https://github.com/cluebotng/component-configs/commits/3652893dce02243971055a6ab740363f103ce104)
* 23:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268031378 (https://github.com/cluebotng/component-configs/commits/31367659ada078f50022f1df4b16b6139db27c09)
* 22:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24267603054 (https://github.com/cluebotng/component-configs/commits/ba252b54cec9387b47dd4ac4a347d4a9c5118c3e)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:53 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442439 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898021 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247620609 (https://github.com/cluebotng/component-configs/commits/e63a941f5b83d97a9751af731c869062ceef4519)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247581365 (https://github.com/cluebotng/component-configs/commits/49becfde53d5f960c8e4df0484cebb2bb4d4c5aa)
* 13:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24246395413 (https://github.com/cluebotng/component-configs/commits/945fa198e64a0e63b777bb570d57d68ef0ce3f69)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245816710 (https://github.com/cluebotng/component-configs/commits/6181fdda40150d3535541f3084ac7ff245f19536)
* 13:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245353001 (https://github.com/cluebotng/component-configs/commits/97eebf1bcdf5be901e0d3fd82c1b3ea6a8668163)
* 13:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245099622 (https://github.com/cluebotng/component-configs/commits/251c10040c01caf2ba9b855050c318d5d2fd8e81)
* 13:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24244959273 (https://github.com/cluebotng/component-configs/commits/2a6605ee2d07c0ff0d690aaa8aabed0ca35bab72)
* 04:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24226442413 (https://github.com/cluebotng/component-configs/commits/4f895f83dae3f356cae2a1bbcfea51dd9d18bd15)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24221486227 (https://github.com/cluebotng/component-configs/commits/d96804861818d7786153d18d47be075a4dbbb6f2)

=== 2026-04-09 ===
* 20:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24212235180 (https://github.com/cluebotng/component-configs/commits/1cd21afab7312bd0122c0e735f8f4dca03019011)
* 19:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24208407251 (https://github.com/cluebotng/component-configs/commits/6cd680dd209bf7fbb01cf24cb6cca82f0fab716d)
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712296 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093220 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)
* 14:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24195131449 (https://github.com/cluebotng/component-configs/commits/6b512f6db7cc4e49078b135e437185906821ae81)

=== 2026-04-08 ===
* 05:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24118645397 (https://github.com/cluebotng/component-configs/commits/908dd70b5972cca0c0dafbe50a0020547b833a4e)

=== 2026-04-07 ===
* 22:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24106708812 (https://github.com/cluebotng/component-configs/commits/b85f56b6997ccf41cc8ea32f33a61809b68b9bc5)

=== 2026-04-02 ===
* 22:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9)

=== 2026-03-31 ===
* 06:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b)

=== 2026-03-27 ===
* 17:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-23 ===
* 10:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640335 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551619 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)

=== 2026-03-20 ===
* 04:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23329334324 (https://github.com/cluebotng/component-configs/commits/bd7700c30291bfab3a656aa8f257292e287a71ca)

=== 2026-03-19 ===
* 09:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23287809902 (https://github.com/cluebotng/component-configs/commits/0976850451c9fbb8c4afb773cc70b91cd7c6fdeb)

=== 2026-03-17 ===
* 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b)

=== 2026-03-13 ===
* 01:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2)

=== 2026-03-10 ===
* 00:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22881598726 (https://github.com/cluebotng/component-configs/commits/bc32d8044077ff83db8b985b87df029ff564ad29)

=== 2026-03-07 ===
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22788149115 (https://github.com/cluebotng/component-configs/commits/b3731fab9a7f4f225ecbe318fa80808de6c904b0)

=== 2026-03-06 ===
* 09:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430)

=== 2026-03-04 ===
* 09:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22662750872 (https://github.com/cluebotng/component-configs/commits/01746ef8804c30c85963ea888a75887ebe879e3b)
* 01:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c)

=== 2026-03-03 ===
* 21:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22642685900 (https://github.com/cluebotng/component-configs/commits/3cbfb68b3c0e7d97130ede1be762389f300234d2)

=== 2026-03-02 ===
* 01:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6)

=== 2026-02-27 ===
* 01:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894)

=== 2026-02-25 ===
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b)

=== 2026-02-24 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22370719324 (https://github.com/cluebotng/component-configs/commits/7170b95a5f9b6be3c928684f1e9c436deb3ddd1f)
* 00:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22331436693 (https://github.com/cluebotng/component-configs/commits/372e84511fdcb0893755ac22f399d3f24f438f7b)

=== 2026-02-21 ===
* 05:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22251332525 (https://github.com/cluebotng/component-configs/commits/29401fbb166f71e375eca7254fe841cb01836d2f)

=== 2026-02-20 ===
* 13:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22226385604 (https://github.com/cluebotng/component-configs/commits/fbd7d861a7062a2c09fd2117cbf569beb53916f4)
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22216991615 (https://github.com/cluebotng/component-configs/commits/64d521535aa35454c28900f70009efc0e9ff4a10)
* 05:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22212024170 (https://github.com/cluebotng/component-configs/commits/9b0508c1c5a875dd795c865e67f2a93d4f247597)

=== 2026-02-19 ===
* 13:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4)
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6)

=== 2026-02-16 ===
* 02:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22047780823 (https://github.com/cluebotng/component-configs/commits/3a1f6b151d38aab4ce1a62509b108ac9afc5230b)

=== 2026-02-15 ===
* 08:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22032600290 (https://github.com/cluebotng/component-configs/commits/80b9cda20d3f21e2f901db6ccbd168bfffb6b063)

=== 2026-02-14 ===
* 20:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023900270 (https://github.com/cluebotng/component-configs/commits/e8878c3f7a08aa1712126c1b6490f6db41621f44)
* 20:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22023724371 (https://github.com/cluebotng/component-configs/commits/de3779f7adea66769077e2380d7b0ce25f3d9e82)
* 20:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023714306 (https://github.com/cluebotng/component-configs/commits/7cd082664340738b5c6cc46d0a195f3814672a3a)
* 19:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022827162 (https://github.com/cluebotng/component-configs/commits/0e1693c2b662aaa0c9264ceef355bcbfbc162ea7)
* 19:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022587196 (https://github.com/cluebotng/component-configs/commits/57dcc675b3ed54fc17f697d6c7b9554b5d06aab0)
* 18:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022444685 (https://github.com/cluebotng/component-configs/commits/1e0b17c59284d25ea8ac39a455abb9921ee6608a)

=== 2025-11-26 ===
* 19:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943)

=== 2025-11-11 ===
* 15:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67)

=== 2025-11-09 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215285730 (https://github.com/cluebotng/component-configs/commits/bf77359fc102b05a026ea8b66dc01ff16a936804)
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215085734 (https://github.com/cluebotng/component-configs/commits/6d8f2491239fbe29d19544922253d9930a88e7a0)
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213996489 (https://github.com/cluebotng/component-configs/commits/38bc77281c9dbd1100915d95ba68705d8a7392a7)
* 20:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213940731 (https://github.com/cluebotng/component-configs/commits/c01b89b7b0455d4f1cc63a2eb002f9c55c0a663f)

=== 2025-11-05 ===
* 19:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f)
* 16:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267282 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-07 ===
* 06:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971)

=== 2025-10-06 ===
* 06:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7)

=== 2025-10-05 ===
* 06:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255184611 (https://github.com/cluebotng/component-configs/commits/7fe1a04069d9d0b4b11019443c85885c202852d4)

=== 2025-10-03 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18215013747 (https://github.com/cluebotng/component-configs/commits/7ab2bbe022e2513dc81a13a7055c4c7736e5f876)

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2)
* 14:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc)
* 09:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)

=== 2025-09-27 ===
* 13:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060157048 (https://github.com/cluebotng/component-configs/commits/3aa079ed0cb7aa29f9ece46a47ad96203e53f242)

=== 2025-09-26 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14)
* 11:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce)

=== 2025-09-25 ===
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c)
* 14:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0)

=== 2025-09-24 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198575 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820837 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009157 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983740752 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17977211158 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:11 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976182663
* 12:06 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976062312
* 06:46 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17968643293

=== 2025-09-22 ===
* 20:43 wmbot~component-configs@tools-bastion: Test migrating log to feed channel
* 20:43 wmbot~damian-scripts@tools-bastion-15: Test migrating log to feed channel
* 19:12 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114

=== 2025-08-24 ===
* 18:02 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.3.0

=== 2025-08-23 ===
* 19:44 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.5
* 18:37 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.4
* 18:10 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.3

=== 2025-08-14 ===
* 13:39 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.2
* 13:27 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.1
* 13:20 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.0

=== 2025-08-13 ===
* 20:07 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.10
* 18:53 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.9

=== 2025-08-11 ===
* 19:49 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.7

=== 2025-08-10 ===
* 18:19 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.6
* 15:06 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.5

=== 2025-08-07 ===
* 15:38 wmbot~damian@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.2

=== 2023-06-15 ===
* 18:43 wm-bot: &lt;root&gt; webservice restart, checked pods


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>eayamq1o81ooqliv4j6sah9pir7g1zi</sha1>
    </revision>
    <revision>
      <id>2478991</id>
      <parentid>2478974</parentid>
      <timestamp>2026-11-07T12:39:04Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856760 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)</comment>
      <origin>2478991</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="59061" sha1="ltori4gzy0lehaitimt91vgh3al8f4d" xml:space="preserve">=== 2026-11-07 ===
* 12:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856760 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819771 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251051 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 16:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695395 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249485663 (https://github.com/cluebotng/component-configs/commits/8c7f8b7d54d8637bd5f421ad2f493b96a7008504)
* 16:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248508387 (https://github.com/cluebotng/component-configs/commits/3acae96966e16f4f9f91683ad36f5e56d329753a)
* 14:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234031860 (https://github.com/cluebotng/component-configs/commits/15c352e208f001eee46b8462013282f7e93d9872)
* 14:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833470 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248930 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188041 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 13:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42223625514 (https://github.com/cluebotng/component-configs/commits/9b07be56cd997c4d5787c311dd3e89941f8e528a)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42222102051 (https://github.com/cluebotng/component-configs/commits/5a9290a457af2e5700d955bc65fbd8d77cad712c)
* 12:21 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485332 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 07:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42181945109 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42177270550 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297558 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42166677986 (https://github.com/cluebotng/component-configs/commits/ceeb81f4d5e5d7b6a4947b95df020093bdea2903)
* 04:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42163050761 (https://github.com/cluebotng/component-configs/commits/82355b13bf40529ca3af84e6749306f7cad42b92)
* 03:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618011 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 15:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42052229859 (https://github.com/cluebotng/component-configs/commits/4f5b065a67c9c1036d100bb6ae2816f193b11b90)
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42049573846 (https://github.com/cluebotng/component-configs/commits/120177aabd7236410755f7187b351195daceb172)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978491372 (https://github.com/cluebotng/component-configs/commits/c13b42c494b46b9b47c56144b4bfef621ce4246b)
* 05:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41976470115 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300319 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 04:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41970294033 (https://github.com/cluebotng/component-configs/commits/249d7f9d3e64c31431d63973b2e535b32af4e788)
* 03:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41961409152 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 23:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942779 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41871475015 (https://github.com/cluebotng/component-configs/commits/2013e4c5bd958fdebe0a0c4511369f5a2457b368)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41777591788 (https://github.com/cluebotng/component-configs/commits/e816c7dcfa9f55dfe3171cd83dcb396d239b99eb)
* 04:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41773053027 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208843 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274459 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119500 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41694203768 (https://github.com/cluebotng/component-configs/commits/50df9beed114de84489c4353abb75acc5cc118bd)
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693690557 (https://github.com/cluebotng/component-configs/commits/2350c7e21a6306b68d1249b6635e7820e90f8d65)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616142 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 06:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264177 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147205 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5)
* 05:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41583533049 (https://github.com/cluebotng/component-configs/commits/417cde41c250a5f0b6f21e72ce00a343ad270295)
* 02:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216266 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434964 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 16:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41220829549 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 14:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074752 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347132 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603064 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529432 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726841 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 09:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186724200 (https://github.com/cluebotng/component-configs/commits/8474642453be7b488acccbcd8e7bfe5261236e6c)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186122222 (https://github.com/cluebotng/component-configs/commits/87121d4752d813dd9d7319b0d336491ee1ee0cb5)
* 08:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002301 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595340 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610852 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626232 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41135430871 (https://github.com/cluebotng/component-configs/commits/118e964d982be74b124949a21f67c966a2b2bea0)
* 21:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558348 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761815 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)
* 19:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41112155068 (https://github.com/cluebotng/component-configs/commits/2e97d5ca05d7bdca41d16a18c30c1820a9b1eb1c)

=== 2026-10-29 ===
* 15:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40903315472 (https://github.com/cluebotng/component-configs/commits/48d12f535986ffcf04bdc857a204667a10e4579d)
* 15:38 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696710 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902023817 (https://github.com/cluebotng/component-configs/commits/d8c689f56957540b9a4fd6c51ce60701a2b483c9)
* 15:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900617605 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898273176 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266172 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)
* 14:48 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40894527337 (https://github.com/cluebotng/component-configs/commits/b6e74433624164c9f93d76d8a23016bfa61d9481)

=== 2026-10-28 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507977 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788346 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987481 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc)
* 05:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879060 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)
* 02:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021329 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf)

=== 2026-10-27 ===
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40547448775 (https://github.com/cluebotng/component-configs/commits/d8e5c2b81841263635a002cdb005aaa7328042b8)
* 16:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112186 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 16:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846798 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 15:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588887 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106)
* 14:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783532 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 14:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784076 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591099 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231000 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21)
* 10:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40491080407 (https://github.com/cluebotng/component-configs/commits/d72ea811ba6c14ea73c3b76fe67040cdac4fa091)
* 09:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261191 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249549 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601965 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed)
* 06:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540119 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457745340 (https://github.com/cluebotng/component-configs/commits/c09fe8cd0f8bc989e3798ba689cbffc5c5cb362a)
* 05:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798133 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721244 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 05:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518090 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765768 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375770 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526960 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183898 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)
* 03:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895976 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40441575561 (https://github.com/cluebotng/component-configs/commits/457585647825bf9845a8c2a5e6ab8dba5aca3763)

=== 2026-08-29 ===
* 12:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33252834244 (https://github.com/cluebotng/component-configs/commits/4cbc4575b18a5c1d868b8284f76bb7ca43fe85d4)
* 10:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33248315658 (https://github.com/cluebotng/component-configs/commits/deff83b4f72b157f24d761127affb1362b599cff)

=== 2026-07-15 ===
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29435050584 (https://github.com/cluebotng/component-configs/commits/912a8ea89e62cddf63a20263572b5c94900ef06a)
* 15:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-02 ===
* 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 20:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28544657044 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861)
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506430 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337269 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 18:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28297170540 (https://github.com/cluebotng/component-configs/commits/12a68f8f80991f77c948817dd50798649df0bfac)
* 17:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28296058232 (https://github.com/cluebotng/component-configs/commits/e1ce8770ee82bd4202c7f54215f5b514b693352d)
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28294935998 (https://github.com/cluebotng/component-configs/commits/dfafa3368a83b4ef570763c77a8e2131ae43e790)
* 16:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28294935998 (https://github.com/cluebotng/component-configs/commits/dfafa3368a83b4ef570763c77a8e2131ae43e790)
* 14:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28292262937 (https://github.com/cluebotng/component-configs/commits/d0d81399db60e997e0f3c84c4bfcd671b88d5e51)

=== 2026-06-24 ===
* 17:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117368498 (https://github.com/cluebotng/component-configs/commits/2a2ca3b7a9ed1a932fd98d3ae6694fc8b386db5d)

=== 2026-06-16 ===
* 17:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849959 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737527 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634025367 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770870 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27562889711 (https://github.com/cluebotng/component-configs/commits/b9c48365007a90ed1c240655306615542751ea6f)
* 17:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27562697218 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2)
* 14:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27554179713 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-14 ===
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27495607670 (https://github.com/cluebotng/component-configs/commits/63eaa8007e638829714e8661427aa311d662b341)

=== 2026-06-13 ===
* 16:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27472937497 (https://github.com/cluebotng/component-configs/commits/38317c72706d30dee2f423e689b46826fa0cbb9a)
* 13:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463970 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)
* 11:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27464936848 (https://github.com/cluebotng/component-configs/commits/d19537391f153a63eba67672cf3aecb76bff362e)

=== 2026-06-11 ===
* 13:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27350281127 (https://github.com/cluebotng/component-configs/commits/41e666d399f22121043fa8132a3e8bf6368e5181)

=== 2026-06-10 ===
* 15:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285125877 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277789050 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276698071 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276511398 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275411814 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339348 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27225356873 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156886 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601200 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)
* 14:04 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27019408986 (https://github.com/cluebotng/component-configs/commits/4de86ac4b524f394ec49dc167d0c75457e981af1)
* 12:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27016160066 (https://github.com/cluebotng/component-configs/commits/3c617707fe44f00c5ac81aa315bfc989a7bd4d00)
* 11:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27012814543 (https://github.com/cluebotng/component-configs/commits/48e7143fc26338766785004996ec66dea5876b0e)
* 11:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27011683531 (https://github.com/cluebotng/component-configs/commits/7b3ba4bae58717e47510c281fdb60f00be6f7c92)

=== 2026-06-01 ===
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978339 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757971428 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864796 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862181 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719681637 (https://github.com/cluebotng/component-configs/commits/02759bc368e1f9901ca399942295a807038321cb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575507 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d)

=== 2026-05-30 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26691257350 (https://github.com/cluebotng/component-configs/commits/1daccd14ff5fe952e32175ee2cf249f2312d99ae)

=== 2026-05-29 ===
* 00:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639469 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20)

=== 2026-05-28 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792853 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8)

=== 2026-05-21 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627549 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056)

=== 2026-05-19 ===
* 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076396 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c)

=== 2026-05-17 ===
* 08:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25727821525 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842068 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562210 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25723480598 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280529 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-06 ===
* 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-02 ===
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392660 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)

=== 2026-04-20 ===
* 23:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1)

=== 2026-04-17 ===
* 00:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422836848 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-11 ===
* 10:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2)
* 10:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1)

=== 2026-04-10 ===
* 23:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268430819 (https://github.com/cluebotng/component-configs/commits/3652893dce02243971055a6ab740363f103ce104)
* 23:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268031378 (https://github.com/cluebotng/component-configs/commits/31367659ada078f50022f1df4b16b6139db27c09)
* 22:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24267603054 (https://github.com/cluebotng/component-configs/commits/ba252b54cec9387b47dd4ac4a347d4a9c5118c3e)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:53 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442439 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898021 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247620609 (https://github.com/cluebotng/component-configs/commits/e63a941f5b83d97a9751af731c869062ceef4519)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247581365 (https://github.com/cluebotng/component-configs/commits/49becfde53d5f960c8e4df0484cebb2bb4d4c5aa)
* 13:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24246395413 (https://github.com/cluebotng/component-configs/commits/945fa198e64a0e63b777bb570d57d68ef0ce3f69)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245816710 (https://github.com/cluebotng/component-configs/commits/6181fdda40150d3535541f3084ac7ff245f19536)
* 13:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245353001 (https://github.com/cluebotng/component-configs/commits/97eebf1bcdf5be901e0d3fd82c1b3ea6a8668163)
* 13:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245099622 (https://github.com/cluebotng/component-configs/commits/251c10040c01caf2ba9b855050c318d5d2fd8e81)
* 13:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24244959273 (https://github.com/cluebotng/component-configs/commits/2a6605ee2d07c0ff0d690aaa8aabed0ca35bab72)
* 04:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24226442413 (https://github.com/cluebotng/component-configs/commits/4f895f83dae3f356cae2a1bbcfea51dd9d18bd15)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24221486227 (https://github.com/cluebotng/component-configs/commits/d96804861818d7786153d18d47be075a4dbbb6f2)

=== 2026-04-09 ===
* 20:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24212235180 (https://github.com/cluebotng/component-configs/commits/1cd21afab7312bd0122c0e735f8f4dca03019011)
* 19:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24208407251 (https://github.com/cluebotng/component-configs/commits/6cd680dd209bf7fbb01cf24cb6cca82f0fab716d)
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712296 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093220 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)
* 14:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24195131449 (https://github.com/cluebotng/component-configs/commits/6b512f6db7cc4e49078b135e437185906821ae81)

=== 2026-04-08 ===
* 05:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24118645397 (https://github.com/cluebotng/component-configs/commits/908dd70b5972cca0c0dafbe50a0020547b833a4e)

=== 2026-04-07 ===
* 22:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24106708812 (https://github.com/cluebotng/component-configs/commits/b85f56b6997ccf41cc8ea32f33a61809b68b9bc5)

=== 2026-04-02 ===
* 22:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9)

=== 2026-03-31 ===
* 06:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b)

=== 2026-03-27 ===
* 17:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-23 ===
* 10:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640335 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551619 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)

=== 2026-03-20 ===
* 04:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23329334324 (https://github.com/cluebotng/component-configs/commits/bd7700c30291bfab3a656aa8f257292e287a71ca)

=== 2026-03-19 ===
* 09:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23287809902 (https://github.com/cluebotng/component-configs/commits/0976850451c9fbb8c4afb773cc70b91cd7c6fdeb)

=== 2026-03-17 ===
* 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b)

=== 2026-03-13 ===
* 01:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2)

=== 2026-03-10 ===
* 00:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22881598726 (https://github.com/cluebotng/component-configs/commits/bc32d8044077ff83db8b985b87df029ff564ad29)

=== 2026-03-07 ===
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22788149115 (https://github.com/cluebotng/component-configs/commits/b3731fab9a7f4f225ecbe318fa80808de6c904b0)

=== 2026-03-06 ===
* 09:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430)

=== 2026-03-04 ===
* 09:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22662750872 (https://github.com/cluebotng/component-configs/commits/01746ef8804c30c85963ea888a75887ebe879e3b)
* 01:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c)

=== 2026-03-03 ===
* 21:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22642685900 (https://github.com/cluebotng/component-configs/commits/3cbfb68b3c0e7d97130ede1be762389f300234d2)

=== 2026-03-02 ===
* 01:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6)

=== 2026-02-27 ===
* 01:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894)

=== 2026-02-25 ===
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b)

=== 2026-02-24 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22370719324 (https://github.com/cluebotng/component-configs/commits/7170b95a5f9b6be3c928684f1e9c436deb3ddd1f)
* 00:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22331436693 (https://github.com/cluebotng/component-configs/commits/372e84511fdcb0893755ac22f399d3f24f438f7b)

=== 2026-02-21 ===
* 05:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22251332525 (https://github.com/cluebotng/component-configs/commits/29401fbb166f71e375eca7254fe841cb01836d2f)

=== 2026-02-20 ===
* 13:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22226385604 (https://github.com/cluebotng/component-configs/commits/fbd7d861a7062a2c09fd2117cbf569beb53916f4)
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22216991615 (https://github.com/cluebotng/component-configs/commits/64d521535aa35454c28900f70009efc0e9ff4a10)
* 05:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22212024170 (https://github.com/cluebotng/component-configs/commits/9b0508c1c5a875dd795c865e67f2a93d4f247597)

=== 2026-02-19 ===
* 13:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4)
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6)

=== 2026-02-16 ===
* 02:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22047780823 (https://github.com/cluebotng/component-configs/commits/3a1f6b151d38aab4ce1a62509b108ac9afc5230b)

=== 2026-02-15 ===
* 08:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22032600290 (https://github.com/cluebotng/component-configs/commits/80b9cda20d3f21e2f901db6ccbd168bfffb6b063)

=== 2026-02-14 ===
* 20:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023900270 (https://github.com/cluebotng/component-configs/commits/e8878c3f7a08aa1712126c1b6490f6db41621f44)
* 20:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22023724371 (https://github.com/cluebotng/component-configs/commits/de3779f7adea66769077e2380d7b0ce25f3d9e82)
* 20:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023714306 (https://github.com/cluebotng/component-configs/commits/7cd082664340738b5c6cc46d0a195f3814672a3a)
* 19:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022827162 (https://github.com/cluebotng/component-configs/commits/0e1693c2b662aaa0c9264ceef355bcbfbc162ea7)
* 19:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022587196 (https://github.com/cluebotng/component-configs/commits/57dcc675b3ed54fc17f697d6c7b9554b5d06aab0)
* 18:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022444685 (https://github.com/cluebotng/component-configs/commits/1e0b17c59284d25ea8ac39a455abb9921ee6608a)

=== 2025-11-26 ===
* 19:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943)

=== 2025-11-11 ===
* 15:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67)

=== 2025-11-09 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215285730 (https://github.com/cluebotng/component-configs/commits/bf77359fc102b05a026ea8b66dc01ff16a936804)
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215085734 (https://github.com/cluebotng/component-configs/commits/6d8f2491239fbe29d19544922253d9930a88e7a0)
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213996489 (https://github.com/cluebotng/component-configs/commits/38bc77281c9dbd1100915d95ba68705d8a7392a7)
* 20:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213940731 (https://github.com/cluebotng/component-configs/commits/c01b89b7b0455d4f1cc63a2eb002f9c55c0a663f)

=== 2025-11-05 ===
* 19:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f)
* 16:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267282 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-07 ===
* 06:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971)

=== 2025-10-06 ===
* 06:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7)

=== 2025-10-05 ===
* 06:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255184611 (https://github.com/cluebotng/component-configs/commits/7fe1a04069d9d0b4b11019443c85885c202852d4)

=== 2025-10-03 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18215013747 (https://github.com/cluebotng/component-configs/commits/7ab2bbe022e2513dc81a13a7055c4c7736e5f876)

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2)
* 14:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc)
* 09:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)

=== 2025-09-27 ===
* 13:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060157048 (https://github.com/cluebotng/component-configs/commits/3aa079ed0cb7aa29f9ece46a47ad96203e53f242)

=== 2025-09-26 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14)
* 11:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce)

=== 2025-09-25 ===
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c)
* 14:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0)

=== 2025-09-24 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198575 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820837 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009157 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983740752 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17977211158 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:11 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976182663
* 12:06 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976062312
* 06:46 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17968643293

=== 2025-09-22 ===
* 20:43 wmbot~component-configs@tools-bastion: Test migrating log to feed channel
* 20:43 wmbot~damian-scripts@tools-bastion-15: Test migrating log to feed channel
* 19:12 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114

=== 2025-08-24 ===
* 18:02 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.3.0

=== 2025-08-23 ===
* 19:44 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.5
* 18:37 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.4
* 18:10 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.3

=== 2025-08-14 ===
* 13:39 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.2
* 13:27 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.1
* 13:20 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.0

=== 2025-08-13 ===
* 20:07 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.10
* 18:53 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.9

=== 2025-08-11 ===
* 19:49 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.7

=== 2025-08-10 ===
* 18:19 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.6
* 15:06 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.5

=== 2025-08-07 ===
* 15:38 wmbot~damian@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.2

=== 2023-06-15 ===
* 18:43 wm-bot: &lt;root&gt; webservice restart, checked pods


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>ltori4gzy0lehaitimt91vgh3al8f4d</sha1>
    </revision>
    <revision>
      <id>2479002</id>
      <parentid>2478991</parentid>
      <timestamp>2026-11-07T12:45:34Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373199788 (https://github.com/cluebotng/component-configs/commits/f66fbfcb1dc21963040e937d86c830cbfd33c4e8)</comment>
      <origin>2479002</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="59267" sha1="aygcof4g4qagz815zixjjfemqbwiuud" xml:space="preserve">=== 2026-11-07 ===
* 12:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373199788 (https://github.com/cluebotng/component-configs/commits/f66fbfcb1dc21963040e937d86c830cbfd33c4e8)
* 12:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856760 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819771 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251051 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 16:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695395 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249485663 (https://github.com/cluebotng/component-configs/commits/8c7f8b7d54d8637bd5f421ad2f493b96a7008504)
* 16:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248508387 (https://github.com/cluebotng/component-configs/commits/3acae96966e16f4f9f91683ad36f5e56d329753a)
* 14:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234031860 (https://github.com/cluebotng/component-configs/commits/15c352e208f001eee46b8462013282f7e93d9872)
* 14:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833470 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248930 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188041 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 13:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42223625514 (https://github.com/cluebotng/component-configs/commits/9b07be56cd997c4d5787c311dd3e89941f8e528a)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42222102051 (https://github.com/cluebotng/component-configs/commits/5a9290a457af2e5700d955bc65fbd8d77cad712c)
* 12:21 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485332 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 07:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42181945109 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42177270550 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297558 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42166677986 (https://github.com/cluebotng/component-configs/commits/ceeb81f4d5e5d7b6a4947b95df020093bdea2903)
* 04:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42163050761 (https://github.com/cluebotng/component-configs/commits/82355b13bf40529ca3af84e6749306f7cad42b92)
* 03:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618011 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 15:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42052229859 (https://github.com/cluebotng/component-configs/commits/4f5b065a67c9c1036d100bb6ae2816f193b11b90)
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42049573846 (https://github.com/cluebotng/component-configs/commits/120177aabd7236410755f7187b351195daceb172)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978491372 (https://github.com/cluebotng/component-configs/commits/c13b42c494b46b9b47c56144b4bfef621ce4246b)
* 05:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41976470115 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300319 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 04:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41970294033 (https://github.com/cluebotng/component-configs/commits/249d7f9d3e64c31431d63973b2e535b32af4e788)
* 03:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41961409152 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 23:47 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942779 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41871475015 (https://github.com/cluebotng/component-configs/commits/2013e4c5bd958fdebe0a0c4511369f5a2457b368)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41777591788 (https://github.com/cluebotng/component-configs/commits/e816c7dcfa9f55dfe3171cd83dcb396d239b99eb)
* 04:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41773053027 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208843 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274459 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119500 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41694203768 (https://github.com/cluebotng/component-configs/commits/50df9beed114de84489c4353abb75acc5cc118bd)
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693690557 (https://github.com/cluebotng/component-configs/commits/2350c7e21a6306b68d1249b6635e7820e90f8d65)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616142 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 06:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264177 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586147205 (https://github.com/cluebotng/component-configs/commits/6295be2043be15b64a1f5a2667c2f9f11b4194b5)
* 05:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41583533049 (https://github.com/cluebotng/component-configs/commits/417cde41c250a5f0b6f21e72ce00a343ad270295)
* 02:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216266 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434964 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 16:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41220829549 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 14:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074752 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347132 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603064 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529432 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726841 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 09:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186724200 (https://github.com/cluebotng/component-configs/commits/8474642453be7b488acccbcd8e7bfe5261236e6c)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186122222 (https://github.com/cluebotng/component-configs/commits/87121d4752d813dd9d7319b0d336491ee1ee0cb5)
* 08:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002301 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595340 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610852 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626232 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41135430871 (https://github.com/cluebotng/component-configs/commits/118e964d982be74b124949a21f67c966a2b2bea0)
* 21:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558348 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761815 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)
* 19:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41112155068 (https://github.com/cluebotng/component-configs/commits/2e97d5ca05d7bdca41d16a18c30c1820a9b1eb1c)

=== 2026-10-29 ===
* 15:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40903315472 (https://github.com/cluebotng/component-configs/commits/48d12f535986ffcf04bdc857a204667a10e4579d)
* 15:38 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696710 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902023817 (https://github.com/cluebotng/component-configs/commits/d8c689f56957540b9a4fd6c51ce60701a2b483c9)
* 15:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900617605 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898273176 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266172 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)
* 14:48 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40894527337 (https://github.com/cluebotng/component-configs/commits/b6e74433624164c9f93d76d8a23016bfa61d9481)

=== 2026-10-28 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741507977 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788346 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40638987481 (https://github.com/cluebotng/component-configs/commits/ed84094285672ba9b9621030959dffcfe3f2fcbc)
* 05:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636879060 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)
* 02:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40621021329 (https://github.com/cluebotng/component-configs/commits/e68d7e50924b4c26fda7a56ae5fbfb784bdf14bf)

=== 2026-10-27 ===
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40547448775 (https://github.com/cluebotng/component-configs/commits/d8e5c2b81841263635a002cdb005aaa7328042b8)
* 16:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112186 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 16:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846798 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 15:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40534588887 (https://github.com/cluebotng/component-configs/commits/a9e566fe0aa786659f3ee0801327ee92e5f26106)
* 14:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783532 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 14:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40517784076 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591099 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 11:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40497231000 (https://github.com/cluebotng/component-configs/commits/ddd9bcc85b24abfbec8521371d7289c73517fb21)
* 10:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40491080407 (https://github.com/cluebotng/component-configs/commits/d72ea811ba6c14ea73c3b76fe67040cdac4fa091)
* 09:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40484261191 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249549 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462601965 (https://github.com/cluebotng/component-configs/commits/b4741b807618f702050fab2dca0bd5bbf12941ed)
* 06:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540119 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457745340 (https://github.com/cluebotng/component-configs/commits/c09fe8cd0f8bc989e3798ba689cbffc5c5cb362a)
* 05:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798133 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721244 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 05:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518090 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765768 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375770 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526960 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183898 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)
* 03:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895976 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40441575561 (https://github.com/cluebotng/component-configs/commits/457585647825bf9845a8c2a5e6ab8dba5aca3763)

=== 2026-08-29 ===
* 12:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33252834244 (https://github.com/cluebotng/component-configs/commits/4cbc4575b18a5c1d868b8284f76bb7ca43fe85d4)
* 10:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33248315658 (https://github.com/cluebotng/component-configs/commits/deff83b4f72b157f24d761127affb1362b599cff)

=== 2026-07-15 ===
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29435050584 (https://github.com/cluebotng/component-configs/commits/912a8ea89e62cddf63a20263572b5c94900ef06a)
* 15:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-02 ===
* 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 20:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28544657044 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861)
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506430 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337269 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 18:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28297170540 (https://github.com/cluebotng/component-configs/commits/12a68f8f80991f77c948817dd50798649df0bfac)
* 17:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28296058232 (https://github.com/cluebotng/component-configs/commits/e1ce8770ee82bd4202c7f54215f5b514b693352d)
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28294935998 (https://github.com/cluebotng/component-configs/commits/dfafa3368a83b4ef570763c77a8e2131ae43e790)
* 16:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28294935998 (https://github.com/cluebotng/component-configs/commits/dfafa3368a83b4ef570763c77a8e2131ae43e790)
* 14:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28292262937 (https://github.com/cluebotng/component-configs/commits/d0d81399db60e997e0f3c84c4bfcd671b88d5e51)

=== 2026-06-24 ===
* 17:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28117368498 (https://github.com/cluebotng/component-configs/commits/2a2ca3b7a9ed1a932fd98d3ae6694fc8b386db5d)

=== 2026-06-16 ===
* 17:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849959 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634737527 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634025367 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770870 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27562889711 (https://github.com/cluebotng/component-configs/commits/b9c48365007a90ed1c240655306615542751ea6f)
* 17:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27562697218 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2)
* 14:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27554179713 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-14 ===
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27495607670 (https://github.com/cluebotng/component-configs/commits/63eaa8007e638829714e8661427aa311d662b341)

=== 2026-06-13 ===
* 16:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27472937497 (https://github.com/cluebotng/component-configs/commits/38317c72706d30dee2f423e689b46826fa0cbb9a)
* 13:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27467463970 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)
* 11:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27464936848 (https://github.com/cluebotng/component-configs/commits/d19537391f153a63eba67672cf3aecb76bff362e)

=== 2026-06-11 ===
* 13:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27350281127 (https://github.com/cluebotng/component-configs/commits/41e666d399f22121043fa8132a3e8bf6368e5181)

=== 2026-06-10 ===
* 15:01 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285125877 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277789050 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276698071 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27276511398 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275411814 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27275339348 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27225356873 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156886 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021601200 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)
* 14:04 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27019408986 (https://github.com/cluebotng/component-configs/commits/4de86ac4b524f394ec49dc167d0c75457e981af1)
* 12:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27016160066 (https://github.com/cluebotng/component-configs/commits/3c617707fe44f00c5ac81aa315bfc989a7bd4d00)
* 11:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27012814543 (https://github.com/cluebotng/component-configs/commits/48e7143fc26338766785004996ec66dea5876b0e)
* 11:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27011683531 (https://github.com/cluebotng/component-configs/commits/7b3ba4bae58717e47510c281fdb60f00be6f7c92)

=== 2026-06-01 ===
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978339 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757971428 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864796 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719862181 (https://github.com/cluebotng/component-configs/commits/8e921d9dd24ae32755f893363b9dfa897cf71c25)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719681637 (https://github.com/cluebotng/component-configs/commits/02759bc368e1f9901ca399942295a807038321cb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719575507 (https://github.com/cluebotng/component-configs/commits/9bb586c7c04b2a5848b2ebc287497a81506f2d1d)

=== 2026-05-30 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26691257350 (https://github.com/cluebotng/component-configs/commits/1daccd14ff5fe952e32175ee2cf249f2312d99ae)

=== 2026-05-29 ===
* 00:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609639469 (https://github.com/cluebotng/component-configs/commits/8c2fccaaae357774084389157d9a305e72eccb20)

=== 2026-05-28 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26592792853 (https://github.com/cluebotng/component-configs/commits/a7971b7e286e177862e5318c40b0d4d868efc7c8)

=== 2026-05-21 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26251627549 (https://github.com/cluebotng/component-configs/commits/96f9184e66a6e4b35a49f02940a213125945b056)

=== 2026-05-19 ===
* 00:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26068076396 (https://github.com/cluebotng/component-configs/commits/f7db7f6fff0d4d6dd451b5f92e75ba755a74129c)

=== 2026-05-17 ===
* 08:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25727821525 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842068 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562210 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25723480598 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280529 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-06 ===
* 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-02 ===
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392660 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)

=== 2026-04-20 ===
* 23:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1)

=== 2026-04-17 ===
* 00:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422836848 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-11 ===
* 10:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2)
* 10:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1)

=== 2026-04-10 ===
* 23:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268430819 (https://github.com/cluebotng/component-configs/commits/3652893dce02243971055a6ab740363f103ce104)
* 23:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268031378 (https://github.com/cluebotng/component-configs/commits/31367659ada078f50022f1df4b16b6139db27c09)
* 22:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24267603054 (https://github.com/cluebotng/component-configs/commits/ba252b54cec9387b47dd4ac4a347d4a9c5118c3e)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:53 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442439 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898021 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247620609 (https://github.com/cluebotng/component-configs/commits/e63a941f5b83d97a9751af731c869062ceef4519)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247581365 (https://github.com/cluebotng/component-configs/commits/49becfde53d5f960c8e4df0484cebb2bb4d4c5aa)
* 13:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24246395413 (https://github.com/cluebotng/component-configs/commits/945fa198e64a0e63b777bb570d57d68ef0ce3f69)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245816710 (https://github.com/cluebotng/component-configs/commits/6181fdda40150d3535541f3084ac7ff245f19536)
* 13:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245353001 (https://github.com/cluebotng/component-configs/commits/97eebf1bcdf5be901e0d3fd82c1b3ea6a8668163)
* 13:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245099622 (https://github.com/cluebotng/component-configs/commits/251c10040c01caf2ba9b855050c318d5d2fd8e81)
* 13:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24244959273 (https://github.com/cluebotng/component-configs/commits/2a6605ee2d07c0ff0d690aaa8aabed0ca35bab72)
* 04:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24226442413 (https://github.com/cluebotng/component-configs/commits/4f895f83dae3f356cae2a1bbcfea51dd9d18bd15)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24221486227 (https://github.com/cluebotng/component-configs/commits/d96804861818d7786153d18d47be075a4dbbb6f2)

=== 2026-04-09 ===
* 20:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24212235180 (https://github.com/cluebotng/component-configs/commits/1cd21afab7312bd0122c0e735f8f4dca03019011)
* 19:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24208407251 (https://github.com/cluebotng/component-configs/commits/6cd680dd209bf7fbb01cf24cb6cca82f0fab716d)
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712296 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093220 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)
* 14:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24195131449 (https://github.com/cluebotng/component-configs/commits/6b512f6db7cc4e49078b135e437185906821ae81)

=== 2026-04-08 ===
* 05:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24118645397 (https://github.com/cluebotng/component-configs/commits/908dd70b5972cca0c0dafbe50a0020547b833a4e)

=== 2026-04-07 ===
* 22:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24106708812 (https://github.com/cluebotng/component-configs/commits/b85f56b6997ccf41cc8ea32f33a61809b68b9bc5)

=== 2026-04-02 ===
* 22:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9)

=== 2026-03-31 ===
* 06:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b)

=== 2026-03-27 ===
* 17:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-23 ===
* 10:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640335 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551619 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)

=== 2026-03-20 ===
* 04:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23329334324 (https://github.com/cluebotng/component-configs/commits/bd7700c30291bfab3a656aa8f257292e287a71ca)

=== 2026-03-19 ===
* 09:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23287809902 (https://github.com/cluebotng/component-configs/commits/0976850451c9fbb8c4afb773cc70b91cd7c6fdeb)

=== 2026-03-17 ===
* 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b)

=== 2026-03-13 ===
* 01:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2)

=== 2026-03-10 ===
* 00:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22881598726 (https://github.com/cluebotng/component-configs/commits/bc32d8044077ff83db8b985b87df029ff564ad29)

=== 2026-03-07 ===
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22788149115 (https://github.com/cluebotng/component-configs/commits/b3731fab9a7f4f225ecbe318fa80808de6c904b0)

=== 2026-03-06 ===
* 09:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430)

=== 2026-03-04 ===
* 09:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22662750872 (https://github.com/cluebotng/component-configs/commits/01746ef8804c30c85963ea888a75887ebe879e3b)
* 01:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c)

=== 2026-03-03 ===
* 21:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22642685900 (https://github.com/cluebotng/component-configs/commits/3cbfb68b3c0e7d97130ede1be762389f300234d2)

=== 2026-03-02 ===
* 01:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6)

=== 2026-02-27 ===
* 01:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894)

=== 2026-02-25 ===
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b)

=== 2026-02-24 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22370719324 (https://github.com/cluebotng/component-configs/commits/7170b95a5f9b6be3c928684f1e9c436deb3ddd1f)
* 00:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22331436693 (https://github.com/cluebotng/component-configs/commits/372e84511fdcb0893755ac22f399d3f24f438f7b)

=== 2026-02-21 ===
* 05:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22251332525 (https://github.com/cluebotng/component-configs/commits/29401fbb166f71e375eca7254fe841cb01836d2f)

=== 2026-02-20 ===
* 13:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22226385604 (https://github.com/cluebotng/component-configs/commits/fbd7d861a7062a2c09fd2117cbf569beb53916f4)
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22216991615 (https://github.com/cluebotng/component-configs/commits/64d521535aa35454c28900f70009efc0e9ff4a10)
* 05:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22212024170 (https://github.com/cluebotng/component-configs/commits/9b0508c1c5a875dd795c865e67f2a93d4f247597)

=== 2026-02-19 ===
* 13:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4)
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6)

=== 2026-02-16 ===
* 02:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22047780823 (https://github.com/cluebotng/component-configs/commits/3a1f6b151d38aab4ce1a62509b108ac9afc5230b)

=== 2026-02-15 ===
* 08:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22032600290 (https://github.com/cluebotng/component-configs/commits/80b9cda20d3f21e2f901db6ccbd168bfffb6b063)

=== 2026-02-14 ===
* 20:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023900270 (https://github.com/cluebotng/component-configs/commits/e8878c3f7a08aa1712126c1b6490f6db41621f44)
* 20:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22023724371 (https://github.com/cluebotng/component-configs/commits/de3779f7adea66769077e2380d7b0ce25f3d9e82)
* 20:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023714306 (https://github.com/cluebotng/component-configs/commits/7cd082664340738b5c6cc46d0a195f3814672a3a)
* 19:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022827162 (https://github.com/cluebotng/component-configs/commits/0e1693c2b662aaa0c9264ceef355bcbfbc162ea7)
* 19:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022587196 (https://github.com/cluebotng/component-configs/commits/57dcc675b3ed54fc17f697d6c7b9554b5d06aab0)
* 18:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022444685 (https://github.com/cluebotng/component-configs/commits/1e0b17c59284d25ea8ac39a455abb9921ee6608a)

=== 2025-11-26 ===
* 19:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943)

=== 2025-11-11 ===
* 15:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67)

=== 2025-11-09 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215285730 (https://github.com/cluebotng/component-configs/commits/bf77359fc102b05a026ea8b66dc01ff16a936804)
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215085734 (https://github.com/cluebotng/component-configs/commits/6d8f2491239fbe29d19544922253d9930a88e7a0)
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213996489 (https://github.com/cluebotng/component-configs/commits/38bc77281c9dbd1100915d95ba68705d8a7392a7)
* 20:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213940731 (https://github.com/cluebotng/component-configs/commits/c01b89b7b0455d4f1cc63a2eb002f9c55c0a663f)

=== 2025-11-05 ===
* 19:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f)
* 16:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267282 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-07 ===
* 06:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971)

=== 2025-10-06 ===
* 06:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7)

=== 2025-10-05 ===
* 06:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255184611 (https://github.com/cluebotng/component-configs/commits/7fe1a04069d9d0b4b11019443c85885c202852d4)

=== 2025-10-03 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18215013747 (https://github.com/cluebotng/component-configs/commits/7ab2bbe022e2513dc81a13a7055c4c7736e5f876)

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2)
* 14:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc)
* 09:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)

=== 2025-09-27 ===
* 13:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060157048 (https://github.com/cluebotng/component-configs/commits/3aa079ed0cb7aa29f9ece46a47ad96203e53f242)

=== 2025-09-26 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14)
* 11:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce)

=== 2025-09-25 ===
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c)
* 14:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0)

=== 2025-09-24 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198575 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820837 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009157 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983740752 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17977211158 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:11 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976182663
* 12:06 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976062312
* 06:46 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17968643293

=== 2025-09-22 ===
* 20:43 wmbot~component-configs@tools-bastion: Test migrating log to feed channel
* 20:43 wmbot~damian-scripts@tools-bastion-15: Test migrating log to feed channel
* 19:12 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114

=== 2025-08-24 ===
* 18:02 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.3.0

=== 2025-08-23 ===
* 19:44 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.5
* 18:37 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.4
* 18:10 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.3

=== 2025-08-14 ===
* 13:39 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.2
* 13:27 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.1
* 13:20 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.0

=== 2025-08-13 ===
* 20:07 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.10
* 18:53 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.9

=== 2025-08-11 ===
* 19:49 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.7

=== 2025-08-10 ===
* 18:19 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.6
* 15:06 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.5

=== 2025-08-07 ===
* 15:38 wmbot~damian@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.2

=== 2023-06-15 ===
* 18:43 wm-bot: &lt;root&gt; webservice restart, checked pods


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>aygcof4g4qagz815zixjjfemqbwiuud</sha1>
    </revision>
  </page>
  <page>
    <title>Tool:Gitlab-account-approval/Log</title>
    <ns>116</ns>
    <id>453906</id>
    <revision>
      <id>2479007</id>
      <parentid>2478905</parentid>
      <timestamp>2026-11-07T22:24:25Z</timestamp>
      <contributor>
        <username>Gitlabaccountapprovalbot</username>
        <id>37332</id>
      </contributor>
      <comment>@nmlt was approved.</comment>
      <origin>2479007</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="70881" sha1="6kjkc3cbig9cxpelv405f1r6ndkthkh" xml:space="preserve">&lt;noinclude&gt;'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__&lt;/noinclude&gt;

=== 2026-11-07 ===
* 22:24 [[gitlab:nmlt|@nmlt]] was approved.

=== 2026-11-06 ===
* 18:27 &quot;hakknorris&quot; was rejected (pending since 2026-08-07T18:26:42.847Z).
* 13:33 [[gitlab:mtozzi|@mtozzi]] was approved.

=== 2026-11-05 ===
* 15:24 &quot;mohammadalawneh&quot; was rejected (pending since 2026-08-06T15:21:47.627Z).

=== 2026-11-03 ===
* 20:39 [[gitlab:mfchris84|@mfchris84]] was approved.

=== 2026-11-01 ===
* 07:51 &quot;neelima25&quot; was rejected (pending since 2026-08-02T07:49:15.913Z).

=== 2026-10-31 ===
* 13:30 [[gitlab:bolbec|@bolbec]] was approved.
* 01:39 [[gitlab:ari|@ari]] was approved.

=== 2026-10-30 ===
* 14:42 &quot;gilfriedman&quot; was rejected (pending since 2026-07-31T14:40:01.677Z).
* 07:45 [[gitlab:immortalance|@immortalance]] was approved.

=== 2026-10-28 ===
* 11:09 &quot;tolland&quot; was rejected (pending since 2026-07-29T11:07:37.797Z).

=== 2026-10-27 ===
* 04:18 [[gitlab:sabrsl|@sabrsl]] was approved.

=== 2026-10-26 ===
* 16:21 [[gitlab:techienk|@techienk]] was approved.
* 00:42 &quot;anupamtadav1&quot; was rejected (pending since 2026-07-27T00:41:50.687Z).

=== 2026-10-25 ===
* 23:36 &quot;anupamyadav&quot; was rejected (pending since 2026-07-26T23:34:58.861Z).
* 19:54 &quot;yoshi&quot; was rejected (pending since 2026-07-26T19:51:54.380Z).
* 02:45 [[gitlab:sdhehua|@sdhehua]] was approved.

=== 2026-10-24 ===
* 03:00 [[gitlab:hariswmid|@hariswmid]] was approved.

=== 2026-10-20 ===
* 14:15 [[gitlab:ryuch|@ryuch]] was approved.
* 14:00 [[gitlab:eudaimonia1|@eudaimonia1]] was approved.
* 10:21 &quot;boyazas&quot; was rejected (pending since 2026-07-21T10:20:02.812Z).

=== 2026-10-18 ===
* 19:36 &quot;prashantpandey&quot; was rejected (pending since 2026-07-19T19:34:19.790Z).
* 06:51 &quot;syrax&quot; was rejected (pending since 2026-07-19T06:50:22.001Z).
* 03:00 [[gitlab:bugreporter|@bugreporter]] was approved.

=== 2026-10-15 ===
* 21:15 &quot;qhairy&quot; was rejected (pending since 2026-07-16T21:13:56.198Z).
* 19:48 &quot;jackblanket847&quot; was rejected (pending since 2026-07-16T19:46:46.355Z).

=== 2026-10-14 ===
* 17:36 &quot;ayushtripathi&quot; was rejected (pending since 2026-07-15T17:33:34.597Z).

=== 2026-10-13 ===
* 09:24 &quot;skranon&quot; was rejected (pending since 2026-07-14T09:23:11.065Z).
* 07:21 [[gitlab:theklan|@theklan]] was approved.

=== 2026-10-12 ===
* 07:30 [[gitlab:blynn|@blynn]] was approved.

=== 2026-10-11 ===
* 04:27 &quot;europepetrol&quot; was rejected (pending since 2026-07-12T04:24:46.926Z).

=== 2026-10-09 ===
* 18:33 [[gitlab:stjn|@stjn]] was approved.
* 14:39 &quot;adarsh2406&quot; was rejected (pending since 2026-07-10T14:39:14.167Z).

=== 2026-10-08 ===
* 09:51 [[gitlab:mathieudenelwmfr|@mathieudenelwmfr]] was approved.
* 00:00 &quot;mcint&quot; was rejected (pending since 2026-07-08T23:58:27.716Z).

=== 2026-10-07 ===
* 20:51 [[gitlab:connieatwork|@connieatwork]] was approved.
* 07:39 [[gitlab:peterxy12|@peterxy12]] was approved.
* 05:15 &quot;chuiimuiiofc&quot; was rejected (pending since 2026-07-08T05:12:33.360Z).

=== 2026-10-06 ===
* 21:21 &quot;giantflightlessbirds&quot; was rejected (pending since 2026-07-07T21:20:24.197Z).

=== 2026-10-05 ===
* 04:18 &quot;manjurul&quot; was rejected (pending since 2026-07-06T04:16:41.067Z).

=== 2026-10-04 ===
* 09:57 &quot;gdh-euwk&quot; was rejected (pending since 2026-07-05T09:55:07.048Z).

=== 2026-10-02 ===
* 14:51 [[gitlab:nichesports|@nichesports]] was approved.
* 12:57 [[gitlab:harii|@harii]] was approved.
* 12:12 &quot;mgagat&quot; was rejected (pending since 2026-07-03T12:10:46.225Z).

=== 2026-09-29 ===
* 13:45 &quot;monathierse&quot; was rejected (pending since 2026-06-30T13:42:41.627Z).
* 00:24 [[gitlab:balance3641|@balance3641]] was approved.

=== 2026-09-27 ===
* 20:12 [[gitlab:brokensegue|@brokensegue]] was approved.
* 04:30 &quot;bharu-123&quot; was rejected (pending since 2026-06-28T04:29:14.159Z).

=== 2026-09-23 ===
* 17:48 &quot;mdh&quot; was rejected (pending since 2026-06-24T17:46:51.726Z).

=== 2026-09-19 ===
* 06:42 &quot;thet1nk3r3r&quot; was rejected (pending since 2026-06-20T06:40:39.066Z).

=== 2026-09-15 ===
* 19:57 &quot;modalya&quot; was rejected (pending since 2026-06-16T19:55:12.189Z).

=== 2026-09-14 ===
* 01:12 [[gitlab:11wb|@11wb]] was approved.

=== 2026-09-13 ===
* 20:54 [[gitlab:mrstarfleetcommand|@mrstarfleetcommand]] was approved.
* 11:36 &quot;nyxira17&quot; was rejected (pending since 2026-06-14T11:35:22.416Z).

=== 2026-09-12 ===
* 19:06 &quot;deuces9ers&quot; was rejected (pending since 2026-06-13T19:04:55.275Z).

=== 2026-09-10 ===
* 08:54 &quot;nehap1706&quot; was rejected (pending since 2026-06-11T08:53:58.819Z).
* 01:48 &quot;tehkittycat&quot; was rejected (pending since 2026-06-11T01:47:45.480Z).

=== 2026-09-09 ===
* 20:12 &quot;lufi2586&quot; was rejected (pending since 2026-06-10T20:09:59.034Z).

=== 2026-09-08 ===
* 02:39 &quot;tantomile&quot; was rejected (pending since 2026-06-09T02:36:22.460Z).

=== 2026-09-07 ===
* 15:45 [[gitlab:dvrted|@dvrted]] was approved.
* 03:00 &quot;jessica22664&quot; was rejected (pending since 2026-06-08T02:58:30.150Z).

=== 2026-09-06 ===
* 21:36 &quot;rp238&quot; was rejected (pending since 2026-06-07T21:33:48.927Z).
* 11:57 &quot;versatiledove231&quot; was rejected (pending since 2026-06-07T11:55:14.557Z).

=== 2026-09-05 ===
* 19:51 &quot;paper9oll&quot; was rejected (pending since 2026-06-06T19:48:46.353Z).
* 04:21 &quot;luminousstaring&quot; was rejected (pending since 2026-06-06T04:20:44.385Z).

=== 2026-09-03 ===
* 08:09 [[gitlab:real68er|@real68er]] was approved.

=== 2026-09-02 ===
* 23:30 [[gitlab:aplucas0703|@aplucas0703]] was approved.
* 16:42 &quot;maximocampo&quot; was rejected (pending since 2026-06-03T16:41:52.538Z).

=== 2026-09-01 ===
* 20:15 &quot;sinaider2000&quot; was rejected (pending since 2026-06-02T20:12:40.894Z).
* 19:03 &quot;t-rt15&quot; was rejected (pending since 2026-06-02T19:02:56.111Z).
* 04:21 &quot;naomi723&quot; was rejected (pending since 2026-06-02T04:20:21.044Z).

=== 2026-08-31 ===
* 20:06 &quot;mccio&quot; was rejected (pending since 2026-06-01T20:05:09.900Z).
* 10:51 [[gitlab:hgzh3|@hgzh3]] was approved.

=== 2026-08-27 ===
* 02:18 [[gitlab:madmax|@madmax]] was approved.

=== 2026-08-26 ===
* 21:42 [[gitlab:mmns21|@mmns21]] was approved.

=== 2026-08-25 ===
* 17:27 [[gitlab:skim|@skim]] was approved.
* 14:57 [[gitlab:wilfredor2|@wilfredor2]] was approved.
* 08:03 [[gitlab:noahmvf|@noahmvf]] was approved.

=== 2026-08-23 ===
* 05:30 &quot;amidewicki&quot; was rejected (pending since 2026-05-24T05:29:29.216Z).
* 01:54 &quot;westcodragen07&quot; was rejected (pending since 2026-05-24T01:53:53.396Z).

=== 2026-08-22 ===
* 15:21 [[gitlab:laraibhasan|@laraibhasan]] was approved.

=== 2026-08-20 ===
* 17:18 [[gitlab:jatwa|@jatwa]] was approved.

=== 2026-08-18 ===
* 11:33 &quot;wladek92&quot; was rejected (pending since 2026-05-19T11:30:32.679Z).

=== 2026-08-17 ===
* 18:39 [[gitlab:erlanger16e|@erlanger16e]] was approved.
* 18:33 &quot;ashishkaranam&quot; was rejected (pending since 2026-05-18T18:32:18.813Z).
* 12:48 [[gitlab:lfs|@lfs]] was approved.

=== 2026-08-16 ===
* 15:42 [[gitlab:pcendrer|@pcendrer]] was approved.

=== 2026-08-15 ===
* 17:33 [[gitlab:flammablepizza|@flammablepizza]] was approved.

=== 2026-08-14 ===
* 13:00 [[gitlab:pacmand|@pacmand]] was approved.

=== 2026-08-12 ===
* 13:21 [[gitlab:doublechek|@doublechek]] was approved.

=== 2026-08-11 ===
* 14:15 [[gitlab:mmilenkovicwmf|@mmilenkovicwmf]] was approved.

=== 2026-08-10 ===
* 17:18 [[gitlab:mustafe|@mustafe]] was approved.
* 10:33 &quot;c8263a20&quot; was rejected (pending since 2026-05-11T10:32:40.353Z).

=== 2026-08-09 ===
* 21:18 [[gitlab:iamnetx|@iamnetx]] was approved.
* 20:09 &quot;yirba&quot; was rejected (pending since 2026-05-10T20:07:07.738Z).
* 06:42 &quot;marsam2489&quot; was rejected (pending since 2026-05-10T06:40:46.276Z).

=== 2026-08-08 ===
* 08:15 [[gitlab:taiwaniajusto|@taiwaniajusto]] was approved.

=== 2026-08-07 ===
* 18:21 [[gitlab:gturkington|@gturkington]] was approved.
* 06:36 &quot;brianbybyby&quot; was rejected (pending since 2026-05-08T06:36:08.459Z).

=== 2026-07-30 ===
* 20:42 &quot;horaciocolbert&quot; was rejected (pending since 2026-04-30T20:41:40.421Z).

=== 2026-07-29 ===
* 10:03 [[gitlab:piastu|@piastu]] was approved.
* 06:33 &quot;rafiul1&quot; was rejected (pending since 2026-04-29T06:33:02.573Z).

=== 2026-07-28 ===
* 16:51 [[gitlab:for-each-next|@for-each-next]] was approved.
* 09:21 [[gitlab:gka|@gka]] was approved.

=== 2026-07-27 ===
* 08:12 [[gitlab:cambob|@cambob]] was approved.

=== 2026-07-26 ===
* 15:39 &quot;demansanaagmailcom&quot; was rejected (pending since 2026-04-26T15:39:03.297Z).

=== 2026-07-24 ===
* 10:54 [[gitlab:ysogo|@ysogo]] was approved.
* 09:39 [[gitlab:pankaj199|@pankaj199]] was approved.

=== 2026-07-23 ===
* 12:30 [[gitlab:fermiboson|@fermiboson]] was approved.
* 09:21 [[gitlab:slashme|@slashme]] was approved.

=== 2026-07-22 ===
* 15:54 [[gitlab:lmedley|@lmedley]] was approved.
* 13:24 &quot;praveen5638&quot; was rejected (pending since 2026-04-22T13:21:23.368Z).
* 12:48 [[gitlab:cyberpower678|@cyberpower678]] was approved.
* 08:30 [[gitlab:nabbegat|@nabbegat]] was approved.
* 08:30 [[gitlab:plyd|@plyd]] was approved.
* 07:06 &quot;ayush8620&quot; was rejected (pending since 2026-04-22T07:03:12.476Z).

=== 2026-07-21 ===
* 15:51 [[gitlab:panieravide|@panieravide]] was approved.
* 15:33 [[gitlab:luisvilla-personal|@luisvilla-personal]] was approved.
* 13:48 [[gitlab:yru|@yru]] was approved.
* 13:09 [[gitlab:deevad|@deevad]] was approved.
* 12:54 [[gitlab:ctdo17|@ctdo17]] was approved.
* 12:36 [[gitlab:jeannenoiraud|@jeannenoiraud]] was approved.
* 12:21 [[gitlab:nadiantara|@nadiantara]] was approved.
* 12:18 [[gitlab:wijltcher|@wijltcher]] was approved.
* 10:33 [[gitlab:nivopol|@nivopol]] was approved.
* 10:30 [[gitlab:johlig|@johlig]] was approved.
* 10:30 [[gitlab:majicita|@majicita]] was approved.
* 10:15 [[gitlab:yongjiapeng|@yongjiapeng]] was approved.
* 10:09 [[gitlab:francyskus|@francyskus]] was approved.
* 09:24 [[gitlab:xanonymusx|@xanonymusx]] was approved.

=== 2026-07-20 ===
* 17:15 &quot;leonidlednev&quot; was rejected (pending since 2026-04-20T17:13:35.108Z).
* 15:27 [[gitlab:rodrigoargenton|@rodrigoargenton]] was approved.
* 05:48 &quot;draftecho&quot; was rejected (pending since 2026-04-20T05:48:06.953Z).

=== 2026-07-19 ===
* 12:21 [[gitlab:boivie|@boivie]] was approved.

=== 2026-07-18 ===
* 16:09 [[gitlab:pharos|@pharos]] was approved.
* 15:45 [[gitlab:priyankar22|@priyankar22]] was approved.
* 15:30 [[gitlab:sisyph|@sisyph]] was approved.

=== 2026-07-13 ===
* 03:45 [[gitlab:dreamyshade|@dreamyshade]] was approved.

=== 2026-07-12 ===
* 09:27 [[gitlab:smk|@smk]] was approved.

=== 2026-07-11 ===
* 14:48 &quot;bigcereal42&quot; was rejected (pending since 2026-04-11T14:47:40.321Z).
* 12:18 &quot;pratyushsawan&quot; was rejected (pending since 2026-04-11T12:16:41.671Z).

=== 2026-07-10 ===
* 14:42 [[gitlab:akaza24|@akaza24]] was approved.
* 12:57 [[gitlab:kormisk|@kormisk]] was approved.

=== 2026-07-07 ===
* 10:36 [[gitlab:olafjanssen|@olafjanssen]] was approved.
* 06:57 &quot;elisapoly-99&quot; was rejected (pending since 2026-04-07T06:55:52.662Z).

=== 2026-07-06 ===
* 11:57 &quot;ma3rouf&quot; was rejected (pending since 2026-04-06T11:56:30.978Z).

=== 2026-07-02 ===
* 15:48 [[gitlab:tekneos|@tekneos]] was approved.

=== 2026-07-01 ===
* 15:12 [[gitlab:mugurolevy|@mugurolevy]] was approved.
* 14:15 [[gitlab:vadymts1|@vadymts1]] was approved.
* 09:57 &quot;mugurolevy&quot; was rejected (pending since 2026-04-01T09:55:19.175Z).

=== 2026-06-30 ===
* 14:27 &quot;shivangisharma&quot; was rejected (pending since 2026-03-31T14:26:44.932Z).

=== 2026-06-29 ===
* 19:03 [[gitlab:thisismattmiller|@thisismattmiller]] was approved.

=== 2026-06-28 ===
* 14:51 &quot;nkwenuinadine&quot; was rejected (pending since 2026-03-29T14:48:32.735Z).
* 14:03 &quot;vaishnavikumbhar&quot; was rejected (pending since 2026-03-29T14:01:30.604Z).
* 13:03 &quot;stepmay&quot; was rejected (pending since 2026-03-29T13:01:59.905Z).
* 06:51 &quot;swallroth&quot; was rejected (pending since 2026-03-29T06:49:54.838Z).

=== 2026-06-26 ===
* 09:39 [[gitlab:lakshita28|@lakshita28]] was approved.
* 07:30 [[gitlab:reeti|@reeti]] was approved.
* 07:30 [[gitlab:anushka10patel|@anushka10patel]] was approved.
* 07:30 &quot;samsaesque&quot; was rejected (pending since 2026-03-27T07:29:57.279Z).
* 05:51 [[gitlab:arpithhhaaa|@arpithhhaaa]] was approved.
* 05:51 [[gitlab:govindlaltl|@govindlaltl]] was approved.

=== 2026-06-25 ===
* 16:09 [[gitlab:sakuraemad|@sakuraemad]] was approved.
* 07:00 &quot;kdh8219&quot; was rejected (pending since 2026-03-26T06:58:05.415Z).

=== 2026-06-24 ===
* 11:54 [[gitlab:sanskardubeydev|@sanskardubeydev]] was approved.
* 10:09 &quot;tanmay789q&quot; was rejected (pending since 2026-03-25T10:07:54.602Z).

=== 2026-06-22 ===
* 19:57 [[gitlab:gouvernathor|@gouvernathor]] was approved.
* 16:45 [[gitlab:lucasbelo|@lucasbelo]] was approved.
* 07:15 &quot;jason2000-cpu&quot; was rejected (pending since 2026-03-23T07:14:09.184Z).

=== 2026-06-21 ===
* 13:18 [[gitlab:egonw|@egonw]] was approved.

=== 2026-06-20 ===
* 10:21 [[gitlab:tways2017|@tways2017]] was approved.

=== 2026-06-19 ===
* 16:06 &quot;wilsonwang2026&quot; was rejected (pending since 2026-03-20T16:06:05.511Z).
* 04:12 [[gitlab:claudio|@claudio]] was approved.

=== 2026-06-18 ===
* 14:21 &quot;royiswariii&quot; was rejected (pending since 2026-03-19T14:19:16.896Z).
* 13:06 [[gitlab:laurabarluzzi|@laurabarluzzi]] was approved.

=== 2026-06-17 ===
* 11:24 &quot;adinathq8x&quot; was rejected (pending since 2026-03-18T11:22:50.098Z).
* 09:45 &quot;nathanveritas&quot; was rejected (pending since 2026-03-18T09:43:51.645Z).

=== 2026-06-15 ===
* 22:39 [[gitlab:mohammadhijjawi|@mohammadhijjawi]] was approved.
* 14:24 &quot;enlisar&quot; was rejected (pending since 2026-03-16T14:23:00.109Z).
* 14:06 &quot;ayaan&quot; was rejected (pending since 2026-03-16T14:03:31.071Z).
* 10:54 &quot;kwametech&quot; was rejected (pending since 2026-03-16T10:54:11.083Z).

=== 2026-06-14 ===
* 17:45 [[gitlab:surajseth520|@surajseth520]] was approved.
* 07:24 &quot;malahimhaseeb&quot; was rejected (pending since 2026-03-15T07:21:57.748Z).

=== 2026-06-11 ===
* 11:48 [[gitlab:cadddr|@cadddr]] was approved.
* 11:18 &quot;wikipiggy&quot; was rejected (pending since 2026-03-12T11:16:09.335Z).
* 07:15 [[gitlab:vesihiisi|@vesihiisi]] was approved.

=== 2026-06-10 ===
* 07:03 [[gitlab:dmiranda|@dmiranda]] was approved.

=== 2026-06-09 ===
* 14:21 [[gitlab:linkgenetic|@linkgenetic]] was approved.
* 14:03 [[gitlab:sjones-ctr|@sjones-ctr]] was approved.
* 12:51 [[gitlab:ekrem|@ekrem]] was approved.

=== 2026-06-08 ===
* 17:48 &quot;jmprax&quot; was rejected (pending since 2026-03-09T17:46:38.807Z).
* 16:15 [[gitlab:ahonc|@ahonc]] was approved.
* 12:57 [[gitlab:rainmonger|@rainmonger]] was approved.

=== 2026-06-07 ===
* 23:03 &quot;shadowthewuff&quot; was rejected (pending since 2026-03-08T23:00:53.442Z).
* 11:45 &quot;wiki-pavan&quot; was rejected (pending since 2026-03-08T11:45:11.116Z).
* 02:39 [[gitlab:launchpad|@launchpad]] was approved.

=== 2026-06-06 ===
* 14:54 &quot;unicord&quot; was rejected (pending since 2026-03-07T14:52:04.992Z).
* 12:48 &quot;chien&quot; was rejected (pending since 2026-03-07T12:48:11.669Z).

=== 2026-06-04 ===
* 14:33 &quot;only-vikas&quot; was rejected (pending since 2026-03-05T14:32:09.186Z).

=== 2026-06-03 ===
* 15:00 [[gitlab:anafibnshahibul|@anafibnshahibul]] was approved.

=== 2026-06-02 ===
* 21:21 &quot;mgagat&quot; was rejected (pending since 2026-03-03T21:18:37.223Z).
* 13:57 &quot;prasunaenumarthy&quot; was rejected (pending since 2026-03-03T13:57:14.847Z).
* 05:48 [[gitlab:tmoney|@tmoney]] was approved.

=== 2026-06-01 ===
* 14:57 &quot;vikram2101&quot; was rejected (pending since 2026-03-02T14:54:26.550Z).
* 12:03 &quot;watshell&quot; was rejected (pending since 2026-03-02T12:03:09.329Z).

=== 2026-05-29 ===
* 12:48 &quot;mounikapotladurthi&quot; was rejected (pending since 2026-02-27T12:45:38.609Z).

=== 2026-05-27 ===
* 20:00 &quot;vinitha&quot; was rejected (pending since 2026-02-25T19:58:43.524Z).
* 16:30 &quot;codeurluce&quot; was rejected (pending since 2026-02-25T16:28:53.973Z).
* 14:33 [[gitlab:thilio|@thilio]] was approved.

=== 2026-05-26 ===
* 12:09 &quot;charisad&quot; was rejected (pending since 2026-02-24T12:07:21.881Z).

=== 2026-05-25 ===
* 22:54 &quot;ddshelto&quot; was rejected (pending since 2026-02-23T22:52:44.427Z).
* 19:51 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).
* 19:48 &quot;lakz-99&quot; was rejected (pending since 2026-02-23T19:47:00.263Z).

=== 2026-05-24 ===
* 18:45 &quot;jiyagupta-cs&quot; was rejected (pending since 2026-02-22T18:43:33.176Z).

=== 2026-05-23 ===
* 13:09 [[gitlab:gauthammohanraj|@gauthammohanraj]] was approved.
* 04:21 [[gitlab:staraction|@staraction]] was approved.

=== 2026-05-22 ===
* 19:03 &quot;i-horich&quot; was rejected (pending since 2026-02-20T19:00:43.519Z).
* 01:48 &quot;50323233&quot; was rejected (pending since 2026-02-20T01:48:05.555Z).

=== 2026-05-21 ===
* 18:51 &quot;kartikeyg0104&quot; was rejected (pending since 2026-02-19T18:48:39.707Z).
* 16:27 [[gitlab:renovatebot|@renovatebot]] was approved.
* 16:06 [[gitlab:gkm563|@gkm563]] was approved.

=== 2026-05-20 ===
* 01:21 &quot;beedellrokejulianlockhart&quot; was rejected (pending since 2026-02-18T01:19:13.284Z).

=== 2026-05-18 ===
* 23:18 &quot;wladek92&quot; was rejected (pending since 2026-02-16T23:16:22.939Z).
* 16:36 [[gitlab:effeietsanders|@effeietsanders]] was approved.

=== 2026-05-14 ===
* 21:00 [[gitlab:nehemienathan|@nehemienathan]] was approved.

=== 2026-05-13 ===
* 10:51 &quot;ssssaaaa&quot; was rejected (pending since 2026-02-11T10:50:36.975Z).

=== 2026-05-12 ===
* 18:06 [[gitlab:psubhashish|@psubhashish]] was approved.
* 08:12 &quot;khan&quot; was rejected (pending since 2026-02-10T08:11:48.776Z).
* 04:27 &quot;galaxysh&quot; was rejected (pending since 2026-02-10T04:24:59.440Z).

=== 2026-05-11 ===
* 12:18 &quot;peterxy12&quot; was rejected (pending since 2026-02-09T12:18:01.982Z).

=== 2026-05-10 ===
* 11:09 &quot;yalihupokn&quot; was rejected (pending since 2026-02-08T11:06:51.336Z).
* 05:12 &quot;wobadha&quot; was rejected (pending since 2026-02-08T05:11:00.569Z).

=== 2026-05-09 ===
* 13:45 &quot;bwiki&quot; was rejected (pending since 2026-02-07T13:43:38.177Z).

=== 2026-05-08 ===
* 09:24 [[gitlab:cwilliams|@cwilliams]] was approved.

=== 2026-05-07 ===
* 14:15 &quot;rehankhan78&quot; was rejected (pending since 2026-02-05T14:13:37.754Z).

=== 2026-05-06 ===
* 11:24 &quot;ari&quot; was rejected (pending since 2026-02-04T11:24:11.760Z).
* 08:09 [[gitlab:neriah|@neriah]] was approved.
* 06:27 [[gitlab:status401|@status401]] was approved.

=== 2026-05-03 ===
* 09:54 [[gitlab:anilk|@anilk]] was approved.

=== 2026-05-02 ===
* 17:54 [[gitlab:sweil|@sweil]] was approved.
* 17:00 [[gitlab:aoppo|@aoppo]] was approved.

=== 2026-05-01 ===
* 21:18 [[gitlab:dawalda|@dawalda]] was approved.

=== 2026-04-30 ===
* 21:42 &quot;merohibine&quot; was rejected (pending since 2026-01-29T21:40:00.756Z).
* 20:54 [[gitlab:tfmorris|@tfmorris]] was approved.
* 17:33 [[gitlab:uyen|@uyen]] was approved.
* 07:39 [[gitlab:mahveotm|@mahveotm]] was approved.
* 06:36 [[gitlab:leo321|@leo321]] was approved.

=== 2026-04-29 ===
* 02:27 [[gitlab:dw31415|@dw31415]] was approved.

=== 2026-04-28 ===
* 23:09 [[gitlab:dtorsani|@dtorsani]] was approved.

=== 2026-04-27 ===
* 23:42 [[gitlab:quinlan|@quinlan]] was approved.
* 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved.

=== 2026-04-26 ===
* 17:36 &quot;kuba-hajnej&quot; was rejected (pending since 2026-01-25T17:33:32.467Z).
* 13:03 &quot;jklamo&quot; was rejected (pending since 2026-01-25T13:02:22.936Z).

=== 2026-04-25 ===
* 20:24 [[gitlab:maldaxura|@maldaxura]] was approved.
* 14:33 [[gitlab:sirtobi|@sirtobi]] was approved.
* 04:18 &quot;ice5678&quot; was rejected (pending since 2026-01-24T04:15:30.008Z).

=== 2026-04-24 ===
* 22:06 [[gitlab:arcstur|@arcstur]] was approved.

=== 2026-04-22 ===
* 23:06 &quot;dtorsani&quot; was rejected (pending since 2026-01-21T23:03:25.843Z).
* 22:18 [[gitlab:egezort|@egezort]] was approved.
* 16:45 &quot;nexpectarpit&quot; was rejected (pending since 2026-01-21T16:43:21.045Z).

=== 2026-04-20 ===
* 19:15 &quot;fitch&quot; was rejected (pending since 2026-01-19T19:12:35.644Z).

=== 2026-04-19 ===
* 02:54 [[gitlab:neoact|@neoact]] was approved.

=== 2026-04-18 ===
* 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved.

=== 2026-04-17 ===
* 13:42 &quot;liselot&quot; was rejected (pending since 2026-01-16T13:39:41.909Z).

=== 2026-04-15 ===
* 17:03 &quot;lahari&quot; was rejected (pending since 2026-01-14T17:02:06.275Z).

=== 2026-04-14 ===
* 13:00 &quot;surajseth520&quot; was rejected (pending since 2026-01-13T12:59:45.906Z).
* 04:51 [[gitlab:canley|@canley]] was approved.
* 01:03 &quot;bshizzle&quot; was rejected (pending since 2026-01-13T01:00:48.120Z).

=== 2026-04-13 ===
* 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved.

=== 2026-04-11 ===
* 12:30 &quot;krithash&quot; was rejected (pending since 2026-01-10T12:27:24.731Z).

=== 2026-04-10 ===
* 15:30 &quot;raunak1709&quot; was rejected (pending since 2026-01-09T15:29:10.901Z).

=== 2026-04-07 ===
* 17:03 [[gitlab:supnabla|@supnabla]] was approved.

=== 2026-04-06 ===
* 20:00 [[gitlab:laerdon|@laerdon]] was approved.
* 19:21 [[gitlab:ljq3|@ljq3]] was approved.

=== 2026-04-04 ===
* 11:06 &quot;mixcc&quot; was rejected (pending since 2026-01-03T11:03:33.922Z).

=== 2026-04-02 ===
* 05:30 [[gitlab:mbh1|@mbh1]] was approved.

=== 2026-04-01 ===
* 18:21 &quot;yuvrajpatil17&quot; was rejected (pending since 2025-12-31T18:20:27.991Z).
* 12:12 [[gitlab:amorii0|@amorii0]] was approved.

=== 2026-03-31 ===
* 11:00 &quot;krrishsehgal&quot; was rejected (pending since 2025-12-30T11:00:16.384Z).

=== 2026-03-30 ===
* 15:36 [[gitlab:atsuko|@atsuko]] was approved.

=== 2026-03-29 ===
* 11:36 [[gitlab:giftcup|@giftcup]] was approved.

=== 2026-03-28 ===
* 14:51 [[gitlab:janeeva1|@janeeva1]] was approved.

=== 2026-03-26 ===
* 13:36 [[gitlab:saiphani02|@saiphani02]] was approved.
* 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved.

=== 2026-03-25 ===
* 09:45 &quot;quansi&quot; was rejected (pending since 2025-12-24T09:42:13.451Z).
* 02:18 [[gitlab:viztor|@viztor]] was approved.

=== 2026-03-24 ===
* 23:18 [[gitlab:maryyann|@maryyann]] was approved.
* 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved.
* 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved.
* 07:39 &quot;fred2675&quot; was rejected (pending since 2025-12-23T07:39:11.380Z).

=== 2026-03-23 ===
* 14:51 [[gitlab:komla|@komla]] was approved.
* 05:51 &quot;lunachuck43&quot; was rejected (pending since 2025-12-22T05:50:17.862Z).
* 04:06 &quot;reza110011&quot; was rejected (pending since 2025-12-22T04:05:25.117Z).

=== 2026-03-20 ===
* 21:54 &quot;mertgor&quot; was rejected (pending since 2025-12-19T21:51:51.419Z).
* 20:57 &quot;autanmahmah&quot; was rejected (pending since 2025-12-19T20:54:51.678Z).
* 09:57 [[gitlab:nethahussain|@nethahussain]] was approved.
* 09:27 [[gitlab:piewriter|@piewriter]] was approved.
* 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved.

=== 2026-03-19 ===
* 21:03 &quot;sayvhior&quot; was rejected (pending since 2025-12-18T21:02:31.699Z).

=== 2026-03-18 ===
* 20:15 [[gitlab:martinmystere|@martinmystere]] was approved.

=== 2026-03-17 ===
* 02:51 &quot;louperivois&quot; was rejected (pending since 2025-12-16T02:50:48.197Z).

=== 2026-03-16 ===
* 12:54 &quot;mokayaj857&quot; was rejected (pending since 2025-12-15T12:53:39.015Z).
* 06:18 &quot;roamer15&quot; was rejected (pending since 2025-12-15T06:16:38.042Z).

=== 2026-03-14 ===
* 11:12 &quot;umaramuhammad&quot; was rejected (pending since 2025-12-13T11:10:44.004Z).
* 09:33 &quot;akuma19&quot; was rejected (pending since 2025-12-13T09:31:39.044Z).
* 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved.

=== 2026-03-12 ===
* 20:24 [[gitlab:11wb|@11wb]] was approved.
* 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved.

=== 2026-03-10 ===
* 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved.

=== 2026-03-06 ===
* 08:09 &quot;vazhayilnewone&quot; was rejected (pending since 2025-12-05T08:07:02.184Z).

=== 2026-03-04 ===
* 20:54 [[gitlab:elphie|@elphie]] was approved.
* 11:39 &quot;ronaldahmed&quot; was rejected (pending since 2025-12-03T11:37:47.492Z).
* 02:12 &quot;ltslw&quot; was rejected (pending since 2025-12-03T02:11:52.040Z).

=== 2026-03-02 ===
* 19:21 &quot;dlopez350&quot; was rejected (pending since 2025-12-01T19:20:38.918Z).
* 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved.

=== 2026-03-01 ===
* 10:51 [[gitlab:clintacc|@clintacc]] was approved.

=== 2026-02-28 ===
* 09:24 &quot;cardboardlamp&quot; was rejected (pending since 2025-11-29T09:22:03.947Z).
* 08:18 &quot;wiki-pavan&quot; was rejected (pending since 2025-11-29T08:16:24.184Z).

=== 2026-02-27 ===
* 20:45 &quot;thisisrick25&quot; was rejected (pending since 2025-11-28T20:42:24.454Z).

=== 2026-02-26 ===
* 13:57 &quot;chuiimuiiofc&quot; was rejected (pending since 2025-11-27T13:57:02.794Z).
* 13:54 &quot;steffpro&quot; was rejected (pending since 2025-11-27T13:52:10.859Z).

=== 2026-02-25 ===
* 21:24 &quot;abubakarhabibudayyabu&quot; was rejected (pending since 2025-11-26T21:22:37.776Z).

=== 2026-02-24 ===
* 05:00 &quot;playboi&quot; was rejected (pending since 2025-11-25T05:00:30.762Z).

=== 2026-02-23 ===
* 14:00 &quot;alph65&quot; was rejected (pending since 2025-11-24T13:59:00.797Z).
* 12:33 [[gitlab:robertsky|@robertsky]] was approved.

=== 2026-02-22 ===
* 00:30 &quot;hp8p&quot; was rejected (pending since 2025-11-23T00:29:24.741Z).

=== 2026-02-19 ===
* 16:45 &quot;clayjar&quot; was rejected (pending since 2025-11-20T16:44:48.380Z).

=== 2026-02-18 ===
* 22:18 &quot;nexus&quot; was rejected (pending since 2025-11-19T22:16:48.818Z).
* 12:00 &quot;bernsteinnn&quot; was rejected (pending since 2025-11-19T11:59:04.427Z).

=== 2026-02-17 ===
* 11:36 &quot;jason2000-cpu&quot; was rejected (pending since 2025-11-18T11:34:00.314Z).

=== 2026-02-16 ===
* 14:54 &quot;smaurya&quot; was rejected (pending since 2025-11-17T14:52:06.906Z).

=== 2026-02-15 ===
* 16:51 &quot;kra-79&quot; was rejected (pending since 2025-11-16T16:50:41.375Z).

=== 2026-02-14 ===
* 15:15 [[gitlab:mess|@mess]] was approved.

=== 2026-02-13 ===
* 13:57 &quot;sopalsuemae957&quot; was rejected (pending since 2025-11-14T13:55:16.921Z).
* 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved.

=== 2026-02-12 ===
* 16:30 &quot;kristinagligoric&quot; was rejected (pending since 2025-11-13T16:29:21.646Z).
* 03:33 [[gitlab:anyehansen|@anyehansen]] was approved.
* 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved.

=== 2026-02-10 ===
* 13:18 [[gitlab:db111|@db111]] was approved.

=== 2026-02-09 ===
* 19:06 &quot;squirrel289&quot; was rejected (pending since 2025-11-10T19:04:27.831Z).

=== 2026-02-06 ===
* 20:54 [[gitlab:gillux|@gillux]] was approved.
* 09:09 [[gitlab:lih|@lih]] was approved.

=== 2026-01-31 ===
* 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved.

=== 2026-01-28 ===
* 14:30 [[gitlab:ademola|@ademola]] was approved.
* 10:51 &quot;watshell&quot; was rejected (pending since 2025-10-29T10:51:01.521Z).

=== 2026-01-26 ===
* 23:06 &quot;tavaresgmg&quot; was rejected (pending since 2025-10-27T23:04:42.140Z).

=== 2026-01-25 ===
* 06:03 &quot;cata&quot; was rejected (pending since 2025-10-26T06:01:26.155Z).

=== 2026-01-24 ===
* 21:15 [[gitlab:wiegels|@wiegels]] was approved.
* 06:30 [[gitlab:blaquans|@blaquans]] was approved.

=== 2026-01-23 ===
* 16:27 [[gitlab:lerickson|@lerickson]] was approved.
* 10:15 &quot;fran0035g&quot; was rejected (pending since 2025-10-24T10:12:17.732Z).

=== 2026-01-22 ===
* 21:00 &quot;hacksyn&quot; was rejected (pending since 2025-10-23T20:59:15.982Z).

=== 2026-01-21 ===
* 17:30 [[gitlab:otcenas11|@otcenas11]] was approved.

=== 2026-01-19 ===
* 21:48 [[gitlab:amdrel|@amdrel]] was approved.
* 04:36 &quot;rayalexa&quot; was rejected (pending since 2025-10-20T04:35:02.094Z).

=== 2026-01-18 ===
* 15:45 &quot;somya&quot; was rejected (pending since 2025-10-19T15:43:43.701Z).
* 06:54 &quot;sergg001&quot; was rejected (pending since 2025-10-19T06:54:12.296Z).

=== 2026-01-16 ===
* 11:57 &quot;zeejohsy&quot; was rejected (pending since 2025-10-17T11:56:22.372Z).
* 04:45 &quot;rocky25&quot; was rejected (pending since 2025-10-17T04:43:33.180Z).

=== 2026-01-15 ===
* 16:39 &quot;tiisu&quot; was rejected (pending since 2025-10-16T16:37:18.438Z).
* 12:00 &quot;noahalorwu&quot; was rejected (pending since 2025-10-16T11:58:26.133Z).
* 10:39 &quot;prjayaiuedu&quot; was rejected (pending since 2025-10-16T10:37:16.947Z).

=== 2026-01-13 ===
* 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved.

=== 2026-01-12 ===
* 17:03 &quot;stagietechs&quot; was rejected (pending since 2025-10-13T17:02:25.281Z).

=== 2026-01-10 ===
* 19:06 &quot;keerthisr&quot; was rejected (pending since 2025-10-11T19:05:01.758Z).

=== 2026-01-09 ===
* 20:36 &quot;lightb&quot; was rejected (pending since 2025-10-10T20:34:20.264Z).

=== 2026-01-08 ===
* 19:42 [[gitlab:tbodt|@tbodt]] was approved.
* 13:57 [[gitlab:martynranyard|@martynranyard]] was approved.

=== 2026-01-07 ===
* 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved.
* 14:27 &quot;dipanshu&quot; was rejected (pending since 2025-10-08T14:26:10.794Z).
* 12:30 &quot;adeolaadesina&quot; was rejected (pending since 2025-10-08T12:29:49.592Z).
* 09:21 &quot;tony-kamande&quot; was rejected (pending since 2025-10-08T09:20:28.421Z).
* 06:18 &quot;hninwuttyi&quot; was rejected (pending since 2025-10-08T06:17:28.006Z).
* 05:09 &quot;andume&quot; was rejected (pending since 2025-10-08T05:07:18.582Z).
* 02:00 &quot;mosope&quot; was rejected (pending since 2025-10-08T01:59:54.800Z).
* 01:15 [[gitlab:tungstalite|@tungstalite]] was approved.

=== 2026-01-06 ===
* 18:24 &quot;leerensucher&quot; was rejected (pending since 2025-10-07T18:21:41.253Z).
* 14:54 &quot;leonidlednev&quot; was rejected (pending since 2025-10-07T14:53:07.273Z).
* 12:57 &quot;alexandre-tingaud&quot; was rejected (pending since 2025-10-07T12:54:27.206Z).

=== 2026-01-04 ===
* 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved.
* 15:18 &quot;makjr&quot; was rejected (pending since 2025-10-05T15:16:31.558Z).
* 14:09 &quot;dakshq&quot; was rejected (pending since 2025-10-05T14:08:40.608Z).

=== 2026-01-03 ===
* 20:42 [[gitlab:apehitkey|@apehitkey]] was approved.
* 18:00 [[gitlab:jeremyb|@jeremyb]] was approved.
* 14:09 [[gitlab:twelephant|@twelephant]] was approved.

=== 2026-01-01 ===
* 11:30 &quot;shellstanislav&quot; was rejected (pending since 2025-10-02T11:29:10.150Z).

=== 2025-12-30 ===
* 19:51 &quot;camilojdiaz&quot; was rejected (pending since 2025-09-30T19:49:24.913Z).

=== 2025-12-29 ===
* 16:03 &quot;zied&quot; was rejected (pending since 2025-09-29T16:01:30.415Z).
* 08:18 &quot;rahulsidpradhan&quot; was rejected (pending since 2025-09-29T08:17:02.849Z).

=== 2025-12-26 ===
* 09:48 &quot;thembo42&quot; was rejected (pending since 2025-09-26T09:45:15.033Z).

=== 2025-12-25 ===
* 14:03 &quot;196936074751&quot; was rejected (pending since 2025-09-25T14:02:31.367Z).

=== 2025-12-23 ===
* 16:21 &quot;ngarnsworthy&quot; was rejected (pending since 2025-09-23T16:20:41.211Z).

=== 2025-12-22 ===
* 12:39 &quot;aza555&quot; was rejected (pending since 2025-09-22T12:38:02.622Z).

=== 2025-12-20 ===
* 23:45 &quot;saph&quot; was rejected (pending since 2025-09-20T23:45:01.222Z).

=== 2025-12-19 ===
* 10:15 &quot;vladdymoses&quot; was rejected (pending since 2025-09-19T10:15:00.999Z).
* 07:15 &quot;dirtylittlepoobah&quot; was rejected (pending since 2025-09-19T07:13:55.537Z).

=== 2025-12-18 ===
* 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved.

=== 2025-12-17 ===
* 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved.
* 18:30 &quot;prudencia&quot; was rejected (pending since 2025-09-17T18:27:18.860Z).
* 02:24 &quot;lottie&quot; was rejected (pending since 2025-09-17T02:21:21.744Z).

=== 2025-12-16 ===
* 09:39 [[gitlab:melcatherine|@melcatherine]] was approved.
* 08:54 [[gitlab:leila237|@leila237]] was approved.

=== 2025-12-15 ===
* 18:27 [[gitlab:royalsailor|@royalsailor]] was approved.
* 09:39 [[gitlab:olaf8940|@olaf8940]] was approved.
* 09:39 &quot;brianbybyby&quot; was rejected (pending since 2025-09-15T09:37:45.430Z).

=== 2025-12-14 ===
* 20:21 [[gitlab:essa237|@essa237]] was approved.
* 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved.

=== 2025-12-13 ===
* 21:54 &quot;mmns21&quot; was rejected (pending since 2025-09-13T21:52:24.017Z).
* 20:33 &quot;bugcrawler&quot; was rejected (pending since 2025-09-13T20:31:09.211Z).

=== 2025-12-12 ===
* 14:39 &quot;ruvchoudhary&quot; was rejected (pending since 2025-09-12T14:36:16.167Z).
* 06:54 &quot;rezadress&quot; was rejected (pending since 2025-09-12T06:52:21.749Z).

=== 2025-12-10 ===
* 17:30 [[gitlab:itsmoon|@itsmoon]] was approved.

=== 2025-12-09 ===
* 15:42 [[gitlab:mercy-o|@mercy-o]] was approved.

=== 2025-12-06 ===
* 16:45 &quot;jacquesradjabu&quot; was rejected (pending since 2025-09-06T16:45:17.969Z).
* 11:27 [[gitlab:ikhitron|@ikhitron]] was approved.

=== 2025-12-01 ===
* 08:12 &quot;halconmilenario21&quot; was rejected (pending since 2025-09-01T08:12:10.262Z).

=== 2025-11-30 ===
* 21:06 [[gitlab:habs|@habs]] was approved.

=== 2025-11-29 ===
* 16:36 &quot;bovimacoco&quot; was rejected (pending since 2025-08-30T16:34:39.712Z).
* 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved.

=== 2025-11-24 ===
* 10:30 &quot;alph65&quot; was rejected (pending since 2025-08-25T10:28:40.957Z).
* 02:24 [[gitlab:yaron|@yaron]] was approved.

=== 2025-11-20 ===
* 16:06 &quot;clayjar&quot; was rejected (pending since 2025-08-21T16:04:54.450Z).

=== 2025-11-17 ===
* 21:09 [[gitlab:ankita97531|@ankita97531]] was approved.

=== 2025-11-16 ===
* 14:15 &quot;commanderkefir&quot; was rejected (pending since 2025-08-17T14:13:14.791Z).
* 08:21 &quot;rehankhan78&quot; was rejected (pending since 2025-08-17T08:19:44.896Z).

=== 2025-11-15 ===
* 14:36 &quot;cyberscribe&quot; was rejected (pending since 2025-08-16T14:34:27.230Z).

=== 2025-11-13 ===
* 04:21 &quot;waddie96&quot; was rejected (pending since 2025-08-14T04:19:27.461Z).

=== 2025-11-11 ===
* 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved.

=== 2025-11-10 ===
* 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved.

=== 2025-11-09 ===
* 22:36 &quot;heinxiety&quot; was rejected (pending since 2025-08-10T22:33:12.041Z).

=== 2025-11-07 ===
* 22:00 [[gitlab:forzagreen|@forzagreen]] was approved.

=== 2025-11-06 ===
* 16:57 [[gitlab:rsilvola|@rsilvola]] was approved.

=== 2025-11-04 ===
* 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved.

=== 2025-11-03 ===
* 17:48 &quot;joewaleed98&quot; was rejected (pending since 2025-08-04T17:46:12.191Z).

=== 2025-11-01 ===
* 18:00 &quot;eliasempresas&quot; was rejected (pending since 2025-08-02T17:58:04.412Z).

=== 2025-10-31 ===
* 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved.
* 04:33 &quot;3ch310n&quot; was rejected (pending since 2025-08-01T04:32:21.982Z).

=== 2025-10-30 ===
* 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved.

=== 2025-10-29 ===
* 14:54 &quot;theap&quot; was rejected (pending since 2025-07-30T14:52:12.066Z).

=== 2025-10-28 ===
* 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved.

=== 2025-10-27 ===
* 07:51 [[gitlab:jmoore111|@jmoore111]] was approved.

=== 2025-10-25 ===
* 21:09 [[gitlab:valor|@valor]] was approved.
* 21:03 [[gitlab:booksmurf|@booksmurf]] was approved.
* 02:48 &quot;mystyc1&quot; was rejected (pending since 2025-07-26T02:46:19.373Z).

=== 2025-10-24 ===
* 05:12 &quot;aadarshmahesh&quot; was rejected (pending since 2025-07-25T05:09:38.264Z).

=== 2025-10-22 ===
* 20:54 [[gitlab:janewanga|@janewanga]] was approved.
* 17:27 &quot;abeljeevan&quot; was rejected (pending since 2025-07-23T17:26:46.884Z).
* 16:12 &quot;shrimpnaur&quot; was rejected (pending since 2025-07-23T16:10:37.864Z).

=== 2025-10-21 ===
* 18:51 &quot;jrmuizel&quot; was rejected (pending since 2025-07-22T18:50:07.315Z).
* 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved.

=== 2025-10-17 ===
* 13:21 [[gitlab:blegodwin|@blegodwin]] was approved.

=== 2025-10-16 ===
* 14:51 [[gitlab:bahago|@bahago]] was approved.
* 14:12 &quot;harikrishna0005&quot; was rejected (pending since 2025-07-17T14:10:48.385Z).
* 14:09 &quot;gauthammohanraj&quot; was rejected (pending since 2025-07-17T14:08:47.643Z).

=== 2025-10-15 ===
* 13:48 [[gitlab:adwivedii|@adwivedii]] was approved.
* 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved.
* 13:03 &quot;childmnajennifer&quot; was rejected (pending since 2025-07-16T13:01:50.236Z).
* 05:06 &quot;vssb4214&quot; was rejected (pending since 2025-07-16T05:05:33.985Z).

=== 2025-10-14 ===
* 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved.
* 15:33 [[gitlab:sadrettin|@sadrettin]] was approved.
* 14:18 [[gitlab:tmwyk|@tmwyk]] was approved.
* 08:42 &quot;yasu0796&quot; was rejected (pending since 2025-07-15T08:41:26.453Z).

=== 2025-10-13 ===
* 16:09 [[gitlab:atlas0007|@atlas0007]] was approved.

=== 2025-10-11 ===
* 17:42 [[gitlab:techwizzie|@techwizzie]] was approved.

=== 2025-10-10 ===
* 19:03 [[gitlab:miiswom|@miiswom]] was approved.
* 16:06 [[gitlab:ninatakang|@ninatakang]] was approved.

=== 2025-10-09 ===
* 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved.
* 14:21 [[gitlab:lebogang|@lebogang]] was approved.
* 14:15 [[gitlab:kimondorose|@kimondorose]] was approved.
* 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved.
* 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved.
* 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved.
* 13:45 [[gitlab:system625|@system625]] was approved.
* 13:45 [[gitlab:rolalove|@rolalove]] was approved.
* 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved.
* 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved.
* 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved.
* 13:36 [[gitlab:aradhya|@aradhya]] was approved.
* 09:57 &quot;bognd&quot; was rejected (pending since 2025-07-10T09:55:48.661Z).

=== 2025-10-08 ===
* 23:36 [[gitlab:sopzy|@sopzy]] was approved.
* 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved.
* 19:39 [[gitlab:levon003|@levon003]] was approved.
* 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved.
* 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved.
* 13:36 [[gitlab:chumzine|@chumzine]] was approved.
* 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved.
* 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved.
* 03:42 &quot;nazhiba&quot; was rejected (pending since 2025-07-09T03:40:12.625Z).
* 02:12 &quot;mafennel&quot; was rejected (pending since 2025-07-09T02:11:40.598Z).

=== 2025-10-07 ===
* 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved.
* 21:30 [[gitlab:rona|@rona]] was approved.
* 21:09 [[gitlab:sandijigs|@sandijigs]] was approved.
* 13:36 &quot;xisbajao&quot; was rejected (pending since 2025-07-08T13:33:35.018Z).
* 01:36 &quot;areczek94&quot; was rejected (pending since 2025-07-08T01:35:40.633Z).

=== 2025-10-06 ===
* 19:21 &quot;wmcarter2017&quot; was rejected (pending since 2025-07-07T19:21:12.899Z).

=== 2025-10-05 ===
* 14:15 &quot;meetmendapara&quot; was rejected (pending since 2025-07-06T14:14:16.726Z).

=== 2025-10-04 ===
* 20:51 &quot;nftbaee&quot; was rejected (pending since 2025-07-05T20:50:57.688Z).

=== 2025-10-03 ===
* 06:12 [[gitlab:javiermonton|@javiermonton]] was approved.

=== 2025-10-02 ===
* 20:15 &quot;talaqalotaibipmp&quot; was rejected (pending since 2025-07-03T20:13:05.164Z).

=== 2025-10-01 ===
* 10:54 &quot;bjensen&quot; was rejected (pending since 2025-07-02T10:53:46.574Z).
* 02:45 &quot;kowal1984&quot; was rejected (pending since 2025-07-02T02:44:56.946Z).

=== 2025-09-30 ===
* 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved.
* 00:24 &quot;adium&quot; was rejected (pending since 2025-07-01T00:23:43.807Z).

=== 2025-09-28 ===
* 08:54 [[gitlab:pexerik|@pexerik]] was approved.

=== 2025-09-27 ===
* 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved.

=== 2025-09-26 ===
* 16:57 &quot;algorithmic&quot; was rejected (pending since 2025-06-27T16:56:17.480Z).
* 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved.
* 13:12 [[gitlab:spushpit|@spushpit]] was approved.

=== 2025-09-20 ===
* 14:06 &quot;bwiki&quot; was rejected (pending since 2025-06-21T13:59:14.749Z).

=== 2025-09-16 ===
* 05:39 [[gitlab:deepchirp|@deepchirp]] was approved.

=== 2025-09-15 ===
* 22:00 [[gitlab:noisk8|@noisk8]] was approved.
* 11:03 &quot;ahonc&quot; was rejected (pending since 2025-06-16T11:00:54.843Z).

=== 2025-09-13 ===
* 18:24 &quot;a-ssh22&quot; was rejected (pending since 2025-06-14T18:23:33.937Z).
* 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved.
* 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved.

=== 2025-09-12 ===
* 17:12 [[gitlab:suyash23|@suyash23]] was approved.
* 00:46 &quot;remotetravel&quot; was rejected (pending since 2025-06-13T00:44:08.171Z).

=== 2025-09-10 ===
* 21:09 &quot;jancborchardt&quot; was rejected (pending since 2025-06-11T21:06:30.759Z).

=== 2025-09-09 ===
* 17:03 [[gitlab:vwf|@vwf]] was approved.
* 06:36 [[gitlab:cactusisme|@cactusisme]] was approved.

=== 2025-09-08 ===
* 18:09 &quot;birushandegeya&quot; was rejected (pending since 2025-06-09T18:08:00.087Z).
* 16:27 &quot;ngarnsworthy&quot; was rejected (pending since 2025-06-09T16:24:37.213Z).
* 12:33 &quot;zolgoyo&quot; was rejected (pending since 2025-06-09T12:31:34.199Z).

=== 2025-09-06 ===
* 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved.

=== 2025-09-05 ===
* 21:45 [[gitlab:sakshi2|@sakshi2]] was approved.
* 20:42 &quot;abdukhaliq1&quot; was rejected (pending since 2025-06-06T20:40:42.023Z).
* 14:27 &quot;beubsamy&quot; was rejected (pending since 2025-06-06T14:27:06.781Z).

=== 2025-09-04 ===
* 23:27 &quot;sdhehua&quot; was rejected (pending since 2025-06-05T23:24:45.777Z).
* 19:00 [[gitlab:perry|@perry]] was approved.
* 11:24 &quot;saintwolf&quot; was rejected (pending since 2025-06-05T11:21:20.176Z).

=== 2025-09-02 ===
* 05:48 [[gitlab:aliu|@aliu]] was approved.

=== 2025-08-29 ===
* 13:30 &quot;kksurendran066&quot; was rejected (pending since 2025-05-30T13:27:48.755Z).

=== 2025-08-28 ===
* 22:18 &quot;tauraamuix&quot; was rejected (pending since 2025-05-29T22:16:08.228Z).

=== 2025-08-26 ===
* 19:03 [[gitlab:dikkulah|@dikkulah]] was approved.

=== 2025-08-22 ===
* 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved.

=== 2025-08-21 ===
* 07:39 [[gitlab:yuka|@yuka]] was approved.

=== 2025-08-19 ===
* 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved.

=== 2025-08-17 ===
* 14:27 &quot;madhan13k&quot; was rejected (pending since 2025-05-18T14:26:08.973Z).

=== 2025-08-15 ===
* 10:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-05-16T10:14:48.403Z).

=== 2025-08-11 ===
* 11:48 &quot;hmmyesbro&quot; was rejected (pending since 2025-05-12T11:45:24.350Z).

=== 2025-08-10 ===
* 13:15 [[gitlab:dactyl|@dactyl]] was approved.

=== 2025-08-09 ===
* 04:39 &quot;xxxx100000&quot; was rejected (pending since 2025-05-10T04:37:44.949Z).

=== 2025-08-08 ===
* 14:33 [[gitlab:josefanthony|@josefanthony]] was approved.

=== 2025-08-07 ===
* 23:42 [[gitlab:robins7|@robins7]] was approved.
* 21:42 [[gitlab:pols12|@pols12]] was approved.
* 17:15 &quot;sbronson&quot; was rejected (pending since 2025-05-08T17:15:08.834Z).
* 14:57 [[gitlab:alvindulle|@alvindulle]] was approved.
* 14:45 [[gitlab:xentos|@xentos]] was approved.
* 06:27 &quot;jamesboste&quot; was rejected (pending since 2025-05-08T06:25:14.793Z).
* 03:57 &quot;ysun&quot; was rejected (pending since 2025-05-08T03:55:07.348Z).

=== 2025-08-06 ===
* 21:51 &quot;pols12&quot; was rejected (pending since 2025-05-07T21:49:13.598Z).
* 01:51 &quot;okeamah&quot; was rejected (pending since 2025-05-07T01:48:50.114Z).

=== 2025-08-05 ===
* 09:15 &quot;mobashir-2013&quot; was rejected (pending since 2025-05-06T09:14:24.069Z).

=== 2025-08-01 ===
* 08:00 &quot;douginamug&quot; was rejected (pending since 2025-05-02T07:57:38.317Z).

=== 2025-07-31 ===
* 02:30 [[gitlab:ads|@ads]] was approved.

=== 2025-07-27 ===
* 13:15 &quot;mrico2703&quot; was rejected (pending since 2025-04-27T13:13:12.346Z).
* 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved.
* 10:17 [[gitlab:fuzzew|@fuzzew]] was approved.
* 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved.
* 05:48 [[gitlab:ecoholic|@ecoholic]] was approved.

=== 2025-07-26 ===
* 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved.
* 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved.
* 11:48 [[gitlab:hridyakk|@hridyakk]] was approved.
* 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved.
* 11:45 [[gitlab:binetaa|@binetaa]] was approved.
* 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved.
* 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved.
* 10:21 [[gitlab:flykrth|@flykrth]] was approved.
* 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved.
* 10:21 [[gitlab:aadhi13|@aadhi13]] was approved.
* 10:21 [[gitlab:malavikam05|@malavikam05]] was approved.
* 10:18 [[gitlab:nf609|@nf609]] was approved.
* 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved.
* 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved.

=== 2025-07-25 ===
* 09:49 [[gitlab:kasyap9|@kasyap9]] was approved.
* 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved.

=== 2025-07-24 ===
* 19:36 [[gitlab:madutgn|@madutgn]] was approved.

=== 2025-07-23 ===
* 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved.

=== 2025-07-22 ===
* 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved.

=== 2025-07-21 ===
* 17:30 [[gitlab:asadiqui|@asadiqui]] was approved.
* 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved.
* 04:27 [[gitlab:damian|@damian]] was approved.

=== 2025-07-20 ===
* 09:42 &quot;mike-khoroshun&quot; was rejected (pending since 2025-04-20T09:42:22.732Z).

=== 2025-07-17 ===
* 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved.
* 13:45 [[gitlab:envlh|@envlh]] was approved.

=== 2025-07-14 ===
* 10:24 [[gitlab:missguru|@missguru]] was approved.
* 00:57 &quot;clarfonthey&quot; was rejected (pending since 2025-04-14T00:56:32.626Z).

=== 2025-07-13 ===
* 01:01 [[gitlab:l235|@l235]] was approved.

=== 2025-07-11 ===
* 03:06 &quot;rodavlas&quot; was rejected (pending since 2025-04-11T03:05:45.590Z).

=== 2025-07-06 ===
* 00:09 &quot;lakasa&quot; was rejected (pending since 2025-04-06T00:06:28.469Z).

=== 2025-07-05 ===
* 21:54 &quot;ctrlzvi&quot; was rejected (pending since 2025-04-05T21:54:12.542Z).
* 14:30 &quot;aminualiyu&quot; was rejected (pending since 2025-04-05T14:27:22.617Z).

=== 2025-07-04 ===
* 03:15 [[gitlab:galstar|@galstar]] was approved.

=== 2025-07-02 ===
* 11:27 &quot;vicolas11&quot; was rejected (pending since 2025-04-02T11:25:12.682Z).

=== 2025-06-29 ===
* 23:12 &quot;naomi723&quot; was rejected (pending since 2025-03-30T23:09:24.630Z).

=== 2025-06-28 ===
* 16:21 &quot;mudeh2372&quot; was rejected (pending since 2025-03-29T16:18:27.057Z).

=== 2025-06-27 ===
* 23:18 &quot;rony143&quot; was rejected (pending since 2025-03-28T23:16:13.671Z).
* 22:21 [[gitlab:rluts|@rluts]] was approved.

=== 2025-06-26 ===
* 13:54 &quot;creativegurus&quot; was rejected (pending since 2025-03-27T13:52:41.706Z).

=== 2025-06-24 ===
* 17:42 [[gitlab:devjadiya|@devjadiya]] was approved.
* 14:00 &quot;dominic-r&quot; was rejected (pending since 2025-03-25T14:00:07.307Z).

=== 2025-06-21 ===
* 00:48 [[gitlab:vriaa|@vriaa]] was approved.

=== 2025-06-18 ===
* 15:21 &quot;ayushkhati1&quot; was rejected (pending since 2025-03-19T15:18:50.062Z).

=== 2025-06-17 ===
* 20:45 &quot;chiomavero&quot; was rejected (pending since 2025-03-18T20:44:13.967Z).
* 00:27 [[gitlab:eggroll97|@eggroll97]] was approved.

=== 2025-06-14 ===
* 20:57 &quot;volvox&quot; was rejected (pending since 2025-03-15T20:56:34.018Z).

=== 2025-06-13 ===
* 16:09 [[gitlab:supergrey|@supergrey]] was approved.
* 11:03 &quot;chqaz&quot; was rejected (pending since 2025-03-14T11:01:09.600Z).
* 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved.
* 10:15 &quot;hearvox&quot; was rejected (pending since 2025-03-14T10:13:13.112Z).

=== 2025-06-12 ===
* 15:18 &quot;jlam&quot; was rejected (pending since 2025-03-13T15:17:54.099Z).

=== 2025-06-09 ===
* 20:48 &quot;dipanjansengupta&quot; was rejected (pending since 2025-03-10T20:48:03.545Z).
* 19:27 [[gitlab:reggycelly|@reggycelly]] was approved.
* 14:51 &quot;arendpieter&quot; was rejected (pending since 2025-03-10T14:51:01.445Z).
* 13:21 [[gitlab:greenreaper|@greenreaper]] was approved.
* 09:33 [[gitlab:mmta|@mmta]] was approved.
* 08:03 &quot;a-ssh22&quot; was rejected (pending since 2025-03-10T08:03:08.111Z).

=== 2025-06-08 ===
* 21:06 &quot;mm-episodenlistedlvaupdater&quot; was rejected (pending since 2025-03-09T21:04:06.323Z).

=== 2025-06-06 ===
* 11:06 [[gitlab:olea|@olea]] was approved.

=== 2025-06-05 ===
* 20:33 [[gitlab:encodedwp|@encodedwp]] was approved.
* 15:00 [[gitlab:toluayo|@toluayo]] was approved.
* 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved.
* 11:54 &quot;sdhehua&quot; was rejected (pending since 2025-03-06T11:51:48.241Z).

=== 2025-06-03 ===
* 21:27 [[gitlab:wewakey|@wewakey]] was approved.
* 12:36 &quot;hunsimon2&quot; was rejected (pending since 2025-03-04T12:34:56.520Z).
* 11:54 &quot;hunsimon&quot; was rejected (pending since 2025-03-04T11:53:54.652Z).

=== 2025-06-02 ===
* 12:01 [[gitlab:jaimedes|@jaimedes]] was approved.

=== 2025-05-30 ===
* 18:00 &quot;sathvik9105&quot; was rejected (pending since 2025-02-28T17:59:42.867Z).
* 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved.
* 10:06 [[gitlab:gpsleo|@gpsleo]] was approved.

=== 2025-05-29 ===
* 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved.

=== 2025-05-28 ===
* 02:57 [[gitlab:saper|@saper]] was approved.

=== 2025-05-27 ===
* 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved.
* 15:33 &quot;satanluimm&quot; was rejected (pending since 2025-02-25T15:32:48.101Z).

=== 2025-05-26 ===
* 23:57 &quot;seyedali220&quot; was rejected (pending since 2025-02-24T23:56:17.621Z).

=== 2025-05-21 ===
* 11:12 [[gitlab:guilherme|@guilherme]] was approved.

=== 2025-05-19 ===
* 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved.

=== 2025-05-18 ===
* 00:00 &quot;xidme&quot; was rejected (pending since 2025-02-15T23:58:56.796Z).

=== 2025-05-17 ===
* 02:39 &quot;kdh8219&quot; was rejected (pending since 2025-02-15T02:36:32.237Z).

=== 2025-05-16 ===
* 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved.

=== 2025-05-15 ===
* 04:30 &quot;inspectorzer0&quot; was rejected (pending since 2025-02-13T04:27:33.179Z).

=== 2025-05-14 ===
* 17:42 [[gitlab:llugo|@llugo]] was approved.

=== 2025-05-13 ===
* 20:18 &quot;mmta&quot; was rejected (pending since 2025-02-11T20:17:23.407Z).

=== 2025-05-11 ===
* 20:51 &quot;jad&quot; was rejected (pending since 2025-02-09T20:49:07.333Z).
* 17:54 &quot;nishchalsundan&quot; was rejected (pending since 2025-02-09T17:52:25.761Z).
* 16:39 &quot;mohammed_abukhadra&quot; was rejected (pending since 2025-02-09T16:39:03.730Z).

=== 2025-05-09 ===
* 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved.

=== 2025-05-08 ===
* 08:18 [[gitlab:mengeditch|@mengeditch]] was approved.

=== 2025-05-07 ===
* 03:45 &quot;xluffy&quot; was rejected (pending since 2025-02-05T03:45:14.181Z).

=== 2025-05-06 ===
* 16:54 &quot;punhaniabhishek&quot; was rejected (pending since 2025-02-04T16:53:50.758Z).
* 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved.

=== 2025-05-02 ===
* 12:24 [[gitlab:tohaomg|@tohaomg]] was approved.
* 11:48 [[gitlab:mavrikant|@mavrikant]] was approved.
* 11:45 [[gitlab:daanvr|@daanvr]] was approved.

=== 2025-05-01 ===
* 09:09 &quot;mjoerg&quot; was rejected (pending since 2025-01-30T09:09:04.204Z).

=== 2025-04-30 ===
* 23:06 &quot;sanskardubey&quot; was rejected (pending since 2025-01-29T23:03:25.489Z).

=== 2025-04-29 ===
* 16:00 &quot;geyslein&quot; was rejected (pending since 2025-01-28T16:00:01.510Z).

=== 2025-04-26 ===
* 09:30 &quot;anjali9027&quot; was rejected (pending since 2025-01-25T09:28:07.064Z).

=== 2025-04-25 ===
* 18:00 &quot;salahhazaa&quot; was rejected (pending since 2025-01-24T17:58:30.030Z).
* 15:15 [[gitlab:yiming|@yiming]] was approved.
* 02:06 &quot;mrchanmp&quot; was rejected (pending since 2025-01-24T02:03:58.308Z).

=== 2025-04-23 ===
* 17:03 &quot;rj2904&quot; was rejected (pending since 2025-01-22T17:03:11.207Z).
* 14:21 &quot;nischay33&quot; was rejected (pending since 2025-01-22T14:19:21.081Z).

=== 2025-04-22 ===
* 19:27 &quot;dj80&quot; was rejected (pending since 2025-01-21T19:25:28.498Z).
* 14:30 [[gitlab:kaimamin|@kaimamin]] was approved.
* 09:57 &quot;debo&quot; was rejected (pending since 2025-01-21T09:54:47.955Z).

=== 2025-04-21 ===
* 12:24 &quot;unshell&quot; was rejected (pending since 2025-01-20T12:21:59.686Z).

=== 2025-04-18 ===
* 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved.

=== 2025-04-16 ===
* 03:09 &quot;dewey&quot; was rejected (pending since 2025-01-15T03:06:17.488Z).

=== 2025-04-15 ===
* 19:45 &quot;emdadul&quot; was rejected (pending since 2025-01-14T19:42:29.285Z).

=== 2025-04-14 ===
* 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved.

=== 2025-04-11 ===
* 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved.

=== 2025-04-10 ===
* 04:12 &quot;bhai420&quot; was rejected (pending since 2025-01-09T04:10:29.430Z).

=== 2025-04-09 ===
* 05:03 &quot;austinvarshney&quot; was rejected (pending since 2025-01-08T05:02:34.175Z).

=== 2025-04-06 ===
* 15:36 [[gitlab:elph|@elph]] was approved.

=== 2025-04-02 ===
* 10:33 [[gitlab:ozge|@ozge]] was approved.

=== 2025-03-31 ===
* 20:15 &quot;demandkey&quot; was rejected (pending since 2024-12-30T20:14:23.096Z).
* 15:18 [[gitlab:danyya|@danyya]] was approved.

=== 2025-03-28 ===
* 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved.
* 15:54 [[gitlab:ilanen1|@ilanen1]] was approved.

=== 2025-03-25 ===
* 19:27 [[gitlab:irfo|@irfo]] was approved.
* 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved.
* 04:33 [[gitlab:paul26|@paul26]] was approved.
* 04:18 &quot;as1100k&quot; was rejected (pending since 2024-12-24T04:18:06.813Z).

=== 2025-03-24 ===
* 11:33 &quot;amzadkhankk&quot; was rejected (pending since 2024-12-23T11:33:14.176Z).

=== 2025-03-23 ===
* 12:24 &quot;wolfdo&quot; was rejected (pending since 2024-12-22T12:23:35.056Z).

=== 2025-03-22 ===
* 09:45 [[gitlab:fjmustak|@fjmustak]] was approved.

=== 2025-03-20 ===
* 18:42 &quot;sathishkokila&quot; was rejected (pending since 2024-12-19T18:39:35.161Z).
* 17:03 [[gitlab:alien4444|@alien4444]] was approved.
* 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved.

=== 2025-03-19 ===
* 22:57 [[gitlab:r1f4t|@r1f4t]] was approved.
* 19:03 &quot;daniel24ps&quot; was rejected (pending since 2024-12-18T19:00:21.249Z).
* 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved.

=== 2025-03-18 ===
* 17:48 &quot;rahulkundu1209&quot; was rejected (pending since 2024-12-17T17:46:41.936Z).
* 08:15 &quot;kirtisikka972&quot; was rejected (pending since 2024-12-17T08:13:25.487Z).

=== 2025-03-15 ===
* 13:30 &quot;tulspal_sidhu&quot; was rejected (pending since 2024-12-14T13:29:10.606Z).
* 01:39 &quot;peacedeadc&quot; was rejected (pending since 2024-12-14T01:37:36.579Z).

=== 2025-03-14 ===
* 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved.
* 02:33 &quot;yxngtrtxll&quot; was rejected (pending since 2024-12-13T02:31:51.658Z).

=== 2025-03-13 ===
* 14:36 [[gitlab:iccander|@iccander]] was approved.

=== 2025-03-12 ===
* 23:21 &quot;jokerchic36&quot; was rejected (pending since 2024-12-11T23:21:00.670Z).
* 15:30 [[gitlab:naomi|@naomi]] was approved.
* 15:27 [[gitlab:cobi|@cobi]] was approved.

=== 2025-03-11 ===
* 12:42 &quot;mohitvermaxx&quot; was rejected (pending since 2024-12-10T12:40:56.967Z).

=== 2025-03-10 ===
* 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved.

=== 2025-03-09 ===
* 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved.
* 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved.

=== 2025-03-07 ===
* 16:54 [[gitlab:hswan|@hswan]] was approved.
* 14:42 [[gitlab:atitkov|@atitkov]] was approved.
* 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved.

=== 2025-03-06 ===
* 17:21 &quot;johnmann&quot; was rejected (pending since 2024-12-05T17:19:24.995Z).

=== 2025-03-05 ===
* 07:33 [[gitlab:monx9494|@monx9494]] was approved.

=== 2025-03-02 ===
* 21:21 &quot;paul26&quot; was rejected (pending since 2024-12-01T21:20:19.681Z).

=== 2025-03-01 ===
* 19:15 [[gitlab:izno|@izno]] was approved.
* 12:45 [[gitlab:nyerho|@nyerho]] was approved.

=== 2025-02-28 ===
* 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved.
* 13:09 &quot;ashwinpraveengo&quot; was rejected (pending since 2024-11-29T13:07:47.240Z).
* 00:18 &quot;eduardoaugusto&quot; was rejected (pending since 2024-11-29T00:17:43.372Z).

=== 2025-02-27 ===
* 20:39 &quot;volkanurl&quot; was rejected (pending since 2024-11-28T20:37:18.101Z).

=== 2025-02-24 ===
* 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved.
* 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved.
* 19:06 [[gitlab:dhardy|@dhardy]] was approved.

=== 2025-02-22 ===
* 19:27 [[gitlab:owuh|@owuh]] was approved.

=== 2025-02-19 ===
* 16:06 [[gitlab:artemkloko|@artemkloko]] was approved.
* 13:03 [[gitlab:jgafnea|@jgafnea]] was approved.

=== 2025-02-17 ===
* 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved.

=== 2025-02-16 ===
* 19:12 &quot;gaurigupta21&quot; was rejected (pending since 2024-11-17T19:11:07.416Z).

=== 2025-02-15 ===
* 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved.

=== 2025-02-14 ===
* 15:21 &quot;nathanbnm&quot; was rejected (pending since 2024-11-15T15:18:19.632Z).

=== 2025-02-13 ===
* 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved.
* 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved.

=== 2025-02-12 ===
* 23:21 &quot;monkeypatch999&quot; was rejected (pending since 2024-11-13T23:20:38.398Z).
* 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved.

=== 2025-02-11 ===
* 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved.

=== 2025-02-09 ===
* 16:15 &quot;mohammed_abukhadra&quot; was rejected (pending since 2024-11-10T16:15:18.361Z).

=== 2025-02-07 ===
* 21:33 &quot;brennan&quot; was rejected (pending since 2024-11-08T21:31:07.351Z).

=== 2025-02-06 ===
* 08:24 &quot;mmta&quot; was rejected (pending since 2024-11-07T08:22:36.724Z).
* 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved.

=== 2025-02-05 ===
* 22:39 &quot;chrissteinchen&quot; was rejected (pending since 2024-11-06T22:38:16.673Z).

=== 2025-02-03 ===
* 07:45 &quot;edriiic&quot; was rejected (pending since 2024-11-04T07:44:46.849Z).
* 01:12 &quot;geppy&quot; was rejected (pending since 2024-11-04T01:10:48.710Z).

=== 2025-02-02 ===
* 13:18 &quot;funa-enpitu&quot; was rejected (pending since 2024-11-03T13:15:46.065Z).

=== 2025-01-31 ===
* 23:42 &quot;nfontes&quot; was rejected (pending since 2024-11-01T23:39:41.755Z).
* 22:51 &quot;sbronson&quot; was rejected (pending since 2024-11-01T22:50:31.871Z).
* 00:42 [[gitlab:farid|@farid]] was approved.

=== 2025-01-27 ===
* 08:15 [[gitlab:eliza189|@eliza189]] was approved.

=== 2025-01-25 ===
* 09:51 [[gitlab:pamputt|@pamputt]] was approved.

=== 2025-01-23 ===
* 14:30 [[gitlab:lubianat|@lubianat]] was approved.
* 11:45 [[gitlab:bootsa|@bootsa]] was approved.

=== 2025-01-21 ===
* 05:09 &quot;niko&quot; was rejected (pending since 2024-07-21T16:10:01.377Z).
* 05:09 &quot;thawizkid369777&quot; was rejected (pending since 2024-07-18T17:42:44.493Z).
* 05:09 &quot;sarthaksingh2&quot; was rejected (pending since 2024-07-10T11:31:30.470Z).
* 05:09 &quot;shriyakt&quot; was rejected (pending since 2024-07-06T04:54:10.248Z).
* 05:09 &quot;akshaya&quot; was rejected (pending since 2024-07-06T04:04:51.488Z).
* 05:09 &quot;alaka03aj&quot; was rejected (pending since 2024-07-05T18:01:54.876Z).
* 05:09 &quot;sulochanaviji-5049&quot; was rejected (pending since 2024-07-01T05:58:00.427Z).
* 05:09 &quot;nayanjnath&quot; was rejected (pending since 2024-07-01T02:51:57.405Z).
* 05:09 &quot;sd44&quot; was rejected (pending since 2024-06-30T04:28:51.436Z).
* 05:09 &quot;metavalent&quot; was rejected (pending since 2024-06-29T01:37:14.210Z).
* 05:09 &quot;wicloudx&quot; was rejected (pending since 2024-06-28T11:51:23.335Z).
* 05:09 &quot;debo&quot; was rejected (pending since 2024-06-28T01:44:59.845Z).
* 05:09 &quot;bwiki&quot; was rejected (pending since 2024-06-23T14:15:38.032Z).
* 05:09 &quot;toprak&quot; was rejected (pending since 2024-06-23T11:35:50.819Z).
* 05:09 &quot;iristeller&quot; was rejected (pending since 2024-06-14T20:53:48.959Z).
* 05:09 &quot;jcolvin&quot; was rejected (pending since 2024-06-12T17:29:01.238Z).
* 05:09 &quot;kalyan&quot; was rejected (pending since 2024-06-07T07:52:46.993Z).
* 05:09 &quot;bluecrystal&quot; was rejected (pending since 2024-06-06T19:16:20.107Z).
* 05:09 &quot;iftttrohit&quot; was rejected (pending since 2024-06-04T12:08:50.818Z).
* 05:09 &quot;pogpotato&quot; was rejected (pending since 2024-06-03T17:58:21.684Z).
* 05:09 &quot;cptlausebaer&quot; was rejected (pending since 2024-05-31T18:53:27.692Z).
* 05:09 &quot;hdevine825&quot; was rejected (pending since 2024-05-31T17:04:18.279Z).
* 05:09 &quot;anaghaa18&quot; was rejected (pending since 2024-05-25T19:14:31.803Z).
* 05:09 &quot;atharvanair04&quot; was rejected (pending since 2024-05-25T14:24:52.825Z).
* 05:09 &quot;anasvemmully&quot; was rejected (pending since 2024-05-25T06:10:27.261Z).
* 05:09 &quot;abhinavmohandas&quot; was rejected (pending since 2024-05-25T06:05:24.825Z).
* 05:09 &quot;kksurendran06&quot; was rejected (pending since 2024-05-25T06:04:38.082Z).
* 05:09 &quot;albertmarshall8896&quot; was rejected (pending since 2024-05-23T09:32:05.462Z).
* 05:09 &quot;akellison&quot; was rejected (pending since 2024-05-17T02:07:24.229Z).
* 05:09 &quot;mainowill&quot; was rejected (pending since 2024-04-16T23:30:33.881Z).
* 05:09 &quot;bzhqc&quot; was rejected (pending since 2024-04-16T19:50:38.676Z).
* 05:09 &quot;safan41&quot; was rejected (pending since 2024-04-16T03:34:48.942Z).
* 05:09 &quot;mgagat&quot; was rejected (pending since 2024-04-16T03:21:51.764Z).
* 05:09 &quot;okeamah&quot; was rejected (pending since 2024-04-16T02:49:00.143Z).
* 05:09 &quot;xuhao61&quot; was rejected (pending since 2024-04-15T23:45:09.083Z).
* 04:47 &quot;cybel&quot; was rejected (pending since 2024-04-15T06:46:35.791Z).

=== 2025-01-20 ===
* 14:33 [[gitlab:your1|@your1]] was approved.

=== 2025-01-18 ===
* 10:09 [[gitlab:galrach600|@galrach600]] was approved.
* 02:51 [[gitlab:blankeclair|@blankeclair]] was approved.

=== 2025-01-17 ===
* 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved.

=== 2025-01-15 ===
* 17:12 [[gitlab:smartse|@smartse]] was approved.

=== 2025-01-14 ===
* 17:03 [[gitlab:naorleizer|@naorleizer]] was approved.

=== 2025-01-13 ===
* 02:45 [[gitlab:wolf20482|@wolf20482]] was approved.

=== 2025-01-12 ===
* 17:45 [[gitlab:tamzin|@tamzin]] was approved.

=== 2025-01-11 ===
* 15:24 [[gitlab:bargioni|@bargioni]] was approved.
* 14:30 [[gitlab:salelya|@salelya]] was approved.
* 10:15 [[gitlab:malakatshy|@malakatshy]] was approved.
* 05:21 [[gitlab:newmcpee|@newmcpee]] was approved.

=== 2025-01-09 ===
* 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved.

=== 2025-01-08 ===
* 16:21 [[gitlab:ukrface|@ukrface]] was approved.

=== 2024-12-28 ===
* 03:27 [[gitlab:twonum|@twonum]] was approved.

=== 2024-12-25 ===
* 06:09 [[gitlab:harsv567|@harsv567]] was approved.

=== 2024-12-21 ===
* 11:24 [[gitlab:amutha2002|@amutha2002]] was approved.

=== 2024-12-20 ===
* 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved.
* 10:00 [[gitlab:ro-shines|@ro-shines]] was approved.
* 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved.

=== 2024-12-18 ===
* 14:45 [[gitlab:soylacarli|@soylacarli]] was approved.

=== 2024-12-16 ===
* 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved.

=== 2024-12-15 ===
* 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved.

=== 2024-12-13 ===
* 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved.

=== 2024-12-10 ===
* 06:39 [[gitlab:ginaan|@ginaan]] was approved.

=== 2024-12-09 ===
* 05:45 [[gitlab:kallinavya|@kallinavya]] was approved.
* 00:54 [[gitlab:viserion-7|@viserion-7]] was approved.

=== 2024-12-08 ===
* 17:27 [[gitlab:wargo|@wargo]] was approved.

=== 2024-12-05 ===
* 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved.

=== 2024-12-02 ===
* 21:21 [[gitlab:a930913|@a930913]] was approved.

=== 2024-12-01 ===
* 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved.

=== 2024-11-21 ===
* 13:45 [[gitlab:sascha|@sascha]] was approved.

=== 2024-11-19 ===
* 16:36 [[gitlab:jly|@jly]] was approved.

=== 2024-11-15 ===
* 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved.

=== 2024-11-14 ===
* 14:15 [[gitlab:stimoroll|@stimoroll]] was approved.

=== 2024-11-09 ===
* 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved.

=== 2024-11-07 ===
* 19:15 [[gitlab:zulf|@zulf]] was approved.
* 05:33 [[gitlab:hassanamin|@hassanamin]] was approved.

=== 2024-11-06 ===
* 19:39 [[gitlab:daniuu|@daniuu]] was approved.
* 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved.

=== 2024-10-09 ===
* 14:45 [[gitlab:jtweed|@jtweed]] was approved.
* 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved.
* 09:06 [[gitlab:wikibayer|@wikibayer]] was approved.

=== 2024-10-06 ===
* 10:27 [[gitlab:keerthan16|@keerthan16]] was approved.

=== 2024-10-04 ===
* 07:45 [[gitlab:hakimi97|@hakimi97]] was approved.

=== 2024-09-30 ===
* 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved.

=== 2024-09-28 ===
* 17:30 [[gitlab:webrunner95|@webrunner95]] was approved.

=== 2024-09-18 ===
* 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved.

=== 2024-09-14 ===
* 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved.

=== 2024-09-06 ===
* 08:48 [[gitlab:mickabarber|@mickabarber]] was approved.

=== 2024-08-27 ===
* 17:36 [[gitlab:edgars|@edgars]] was approved.

=== 2024-08-22 ===
* 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved.

=== 2024-08-14 ===
* 19:21 [[gitlab:jfk|@jfk]] was approved.

=== 2024-08-13 ===
* 17:57 [[gitlab:daxserver|@daxserver]] was approved.

=== 2024-08-11 ===
* 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved.

=== 2024-08-10 ===
* 08:42 [[gitlab:ashig|@ashig]] was approved.

=== 2024-08-09 ===
* 14:09 [[gitlab:masssly|@masssly]] was approved.

=== 2024-08-05 ===
* 22:15 [[gitlab:mrtortue|@mrtortue]] was approved.

=== 2024-08-02 ===
* 16:21 [[gitlab:dsantini|@dsantini]] was approved.

=== 2024-07-31 ===
* 11:54 [[gitlab:cptviraj|@cptviraj]] was approved.

=== 2024-07-30 ===
* 19:09 [[gitlab:iniquity|@iniquity]] was approved.
* 10:00 [[gitlab:collins|@collins]] was approved.

=== 2024-07-27 ===
* 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved.

=== 2024-07-25 ===
* 12:36 [[gitlab:mszabo|@mszabo]] was approved.
* 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved.

=== 2024-07-24 ===
* 08:05 [[gitlab:dragoniez|@dragoniez]] was approved.

=== 2024-07-23 ===
* 06:54 [[gitlab:mirji|@mirji]] was approved.

=== 2024-07-16 ===
* 10:00 [[gitlab:lakejason0|@lakejason0]] was approved.

=== 2024-07-12 ===
* 11:33 [[gitlab:cn|@cn]] was approved.
* 08:12 [[gitlab:unchampignon|@unchampignon]] was approved.

=== 2024-07-07 ===
* 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved.
* 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved.

=== 2024-07-06 ===
* 11:18 [[gitlab:dibya|@dibya]] was approved.
* 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved.

=== 2024-07-05 ===
* 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved.

=== 2024-07-02 ===
* 19:00 [[gitlab:ebrahim|@ebrahim]] was approved.

=== 2024-07-01 ===
* 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved.
* 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved.

=== 2024-06-30 ===
* 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved.
* 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved.
* 06:30 [[gitlab:rihaan180|@rihaan180]] was approved.

=== 2024-06-27 ===
* 17:36 [[gitlab:driedmueller|@driedmueller]] was approved.

=== 2024-06-19 ===
* 12:57 [[gitlab:audreypenven|@audreypenven]] was approved.

=== 2024-06-16 ===
* 01:18 [[gitlab:roysmith|@roysmith]] was approved.

=== 2024-06-08 ===
* 02:45 [[gitlab:jleedev|@jleedev]] was approved.

=== 2024-06-03 ===
* 13:57 [[gitlab:afeder|@afeder]] was approved.

=== 2024-06-01 ===
* 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved.

=== 2024-05-30 ===
* 16:42 [[gitlab:krlsca|@krlsca]] was approved.

=== 2024-05-28 ===
* 11:24 [[gitlab:rickijay|@rickijay]] was approved.

=== 2024-05-26 ===
* 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved.

=== 2024-05-25 ===
* 07:24 [[gitlab:jony|@jony]] was approved.

=== 2024-05-23 ===
* 08:45 [[gitlab:lepticed7|@lepticed7]] was approved.

=== 2024-05-22 ===
* 20:42 [[gitlab:echecs|@echecs]] was approved.

=== 2024-05-21 ===
* 13:33 [[gitlab:mbs|@mbs]] was approved.

=== 2024-05-19 ===
* 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved.

=== 2024-05-18 ===
* 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved.

=== 2024-05-17 ===
* 08:54 [[gitlab:grapedog|@grapedog]] was approved.

=== 2024-05-08 ===
* 19:42 [[gitlab:kelhurd|@kelhurd]] was approved.
* 19:06 [[gitlab:khurd|@khurd]] was approved.

=== 2024-05-06 ===
* 19:48 [[gitlab:j3j5|@j3j5]] was approved.
* 12:06 [[gitlab:tk-999|@tk-999]] was approved.

=== 2024-05-05 ===
* 22:09 [[gitlab:pppery|@pppery]] was approved.
* 20:33 [[gitlab:sakretsu|@sakretsu]] was approved.
* 12:12 [[gitlab:waterquark|@waterquark]] was approved.

=== 2024-05-04 ===
* 09:03 [[gitlab:multichill|@multichill]] was approved.
* 07:42 [[gitlab:abaris|@abaris]] was approved.

=== 2024-05-03 ===
* 14:57 [[gitlab:maurusian|@maurusian]] was approved.

=== 2024-04-24 ===
* 05:48 [[gitlab:wolfinux|@wolfinux]] was approved.

=== 2024-04-23 ===
* 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved.

=== 2024-04-21 ===
* 06:51 [[gitlab:alon|@alon]] was approved.

=== 2024-04-17 ===
* 23:33 [[gitlab:derenrich|@derenrich]] was approved.

=== 2024-04-16 ===
* 17:18 [[gitlab:valcio|@valcio]] was approved.

=== 2024-04-14 ===
* 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved.

=== 2024-04-06 ===
* 12:48 [[gitlab:theprotonade|@theprotonade]] was approved.

=== 2024-04-02 ===
* 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved.

=== 2024-03-30 ===
* 13:36 [[gitlab:lpintscher|@lpintscher]] was approved.

=== 2024-03-26 ===
* 17:09 [[gitlab:eenabulele|@eenabulele]] was approved.

=== 2024-03-25 ===
* 14:27 [[gitlab:tuukka|@tuukka]] was approved.

=== 2024-03-24 ===
* 12:24 [[gitlab:firefly|@firefly]] was approved.

=== 2024-03-21 ===
* 19:33 [[gitlab:universal-omega|@universal-omega]] was approved.

=== 2024-03-17 ===
* 10:36 [[gitlab:bisel91|@bisel91]] was approved.

=== 2024-03-16 ===
* 10:09 [[gitlab:delord|@delord]] was approved.
* 00:42 [[gitlab:athulvis1|@athulvis1]] was approved.

=== 2024-03-15 ===
* 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved.
* 08:30 [[gitlab:peachey88|@peachey88]] was approved.
* 06:51 [[gitlab:derick|@derick]] was approved.

=== 2024-03-12 ===
* 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved.

=== 2024-03-06 ===
* 13:21 [[gitlab:desianabae1|@desianabae1]] was approved.

=== 2024-03-05 ===
* 19:21 [[gitlab:ep1c|@ep1c]] was approved.
* 16:33 [[gitlab:jasmine|@jasmine]] was approved.

=== 2024-03-02 ===
* 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved.

=== 2024-02-29 ===
* 23:18 [[gitlab:arandomname123|@arandomname123]] was approved.
* 18:03 [[gitlab:baba|@baba]] was approved.
* 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved.
* 03:09 [[gitlab:sds|@sds]] was approved.

=== 2024-02-27 ===
* 23:33 [[gitlab:lofhi|@lofhi]] was approved.

=== 2024-02-15 ===
* 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved.

=== 2024-02-14 ===
* 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved.

=== 2024-02-13 ===
* 13:06 [[gitlab:dringsim|@dringsim]] was approved.

=== 2024-02-12 ===
* 17:36 [[gitlab:haak|@haak]] was approved.

=== 2024-02-05 ===
* 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved.
* 17:14 [[gitlab:ahecht|@ahecht]] was approved.

=== 2024-02-01 ===
* 09:27 [[gitlab:arinaigum|@arinaigum]] was approved.
* 00:15 [[gitlab:jas42|@jas42]] was approved.
* 00:15 [[gitlab:edhu|@edhu]] was approved.
* 00:15 [[gitlab:marnanel|@marnanel]] was approved.
* 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved.
* 00:15 [[gitlab:amasotti|@amasotti]] was approved.
* 00:15 [[gitlab:deni|@deni]] was approved.
* 00:15 [[gitlab:cyber|@cyber]] was approved.
* 00:15 [[gitlab:saroj|@saroj]] was approved.

=== 2024-01-29 ===
* 21:42 [[gitlab:rgupta|@rgupta]] was approved.

=== 2024-01-07 ===
* 09:48 [[gitlab:lutrome|@lutrome]] was approved.

=== 2024-01-05 ===
* 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved.
* 02:51 [[gitlab:braunobruno|@braunobruno]] was approved.
* 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved.
* 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved.

=== 2024-01-03 ===
* 14:45 [[gitlab:gabina|@gabina]] was approved.

=== 2024-01-02 ===
* 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved.

=== 2023-12-23 ===
* 00:33 [[gitlab:aram|@aram]] was approved.

=== 2023-12-22 ===
* 16:24 [[gitlab:elpitareio|@elpitareio]] was approved.

=== 2023-12-21 ===
* 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved.
* 00:43 [[gitlab:ederporto|@ederporto]] was approved.
* 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved.
* 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved.
* 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved.
* 00:43 [[gitlab:air7538|@air7538]] was approved.
* 00:43 [[gitlab:anzx|@anzx]] was approved.
* 00:43 [[gitlab:tekask1903|@tekask1903]] was approved.
* 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved.
* 00:42 [[gitlab:mpaa|@mpaa]] was approved.
* 00:42 [[gitlab:kutay|@kutay]] was approved.
* 00:42 [[gitlab:wattmto|@wattmto]] was approved.</text>
      <sha1>6kjkc3cbig9cxpelv405f1r6ndkthkh</sha1>
    </revision>
  </page>
  <page>
    <title>User:Max</title>
    <ns>2</ns>
    <id>454410</id>
    <revision>
      <id>2479006</id>
      <parentid>2449401</parentid>
      <timestamp>2026-11-07T16:24:06Z</timestamp>
      <contributor>
        <username>Max</username>
        <id>37491</id>
      </contributor>
      <minor />
      <comment>upd ([[:m:User:Max/MaxEdit.js|MaxEdit]])</comment>
      <origin>2479006</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="94" sha1="lekyp0ur9fjnmivqbsi74kqwxlwtrjv" xml:space="preserve">{{#babel:hi-4|en-4}}
[[:meta:User:Max]]

== Contact me ==
* Discord: madmax.wp
* IRC: MadMaxWP</text>
      <sha1>lekyp0ur9fjnmivqbsi74kqwxlwtrjv</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-staging/SAL</title>
    <ns>498</ns>
    <id>458787</id>
    <revision>
      <id>2478983</id>
      <parentid>2478855</parentid>
      <timestamp>2026-11-07T12:35:02Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819817 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478983</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="28771" sha1="brait0uoklx6vdahsotyt41krvnnu2t" xml:space="preserve">=== 2026-11-07 ===
* 12:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819817 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250564899 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248723213 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833553 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233691092 (https://github.com/cluebotng/component-configs/commits/b630be8dad4a568f64c5a4b612248b3a15b220a3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248999 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224187997 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485410 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42200307291 (https://github.com/cluebotng/component-configs/commits/255eec1d87ca571dcfd0f09253a9504e7d2b70f1)
* 07:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181945089 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297535 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 04:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42162423773 (https://github.com/cluebotng/component-configs/commits/f0d701fcec7061cd6d2bf7ea127a73895e4a005f)

=== 2026-11-05 ===
* 05:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41976470140 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974543465 (https://github.com/cluebotng/component-configs/commits/7f5f130ad27cc6b58d18dc7db9fb6f17a994795f)
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300445 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214249 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107416 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208797 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274531 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064637 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2)

=== 2026-11-03 ===
* 18:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580691 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616232 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215680 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264114 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696321 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216275 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)

=== 2026-11-01 ===
* 21:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41350442836 (https://github.com/cluebotng/component-configs/commits/9f8d02aa99dd4b425755d1cf842fbce1fa045266)
* 12:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41306290028 (https://github.com/cluebotng/component-configs/commits/d2705aba98618c9fa1aecc8578e6861ed2ef9ef8)

=== 2026-10-31 ===
* 14:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074738 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347156 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603045 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529472 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 10:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191380260 (https://github.com/cluebotng/component-configs/commits/2be620aa8d34f581c9c9bb72401e2bcb82705c7d)

=== 2026-10-30 ===
* 22:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41139892792 (https://github.com/cluebotng/component-configs/commits/ce1c920d2a60015846167ad5fba607bfa99e27fa)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610878 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626139 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:48 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761792 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)

=== 2026-10-29 ===
* 15:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902680991 (https://github.com/cluebotng/component-configs/commits/da88cb5e1e9693171a65562139a9fbf29fa65dfe)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902241168 (https://github.com/cluebotng/component-configs/commits/77ffbc8ebcebe0ba0b96808c0324623bd61aadf9)
* 15:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676116 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757791 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898265897 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)
* 14:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40894578776 (https://github.com/cluebotng/component-configs/commits/d180ba6310a8569d7c091af3c9dd92dcdbda1991)

=== 2026-10-28 ===
* 18:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217679 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe)
* 18:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741508078 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788367 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636878933 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)

=== 2026-10-27 ===
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112336 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 14:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783500 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 13:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108521 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591043 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 08:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249856 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 05:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721145 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518212 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826228 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9)
* 04:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765837 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375498 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527051 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895988 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736579 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6)

=== 2026-10-26 ===
* 23:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679333 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a)

=== 2026-09-01 ===
* 19:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856145 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3)

=== 2026-07-13 ===
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29258212925 (https://github.com/cluebotng/component-configs/commits/2d491fa13ec950460297fa6cfae6801b75d17cf4)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412753 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506690 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804682 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 11:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27615770915 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 14:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179699 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-13 ===
* 13:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27467463967 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285127629 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27280628230 (https://github.com/cluebotng/component-configs/commits/9b8bd8bb539a243486f5bdbbee92001379161762)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788856 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276642792 (https://github.com/cluebotng/component-configs/commits/0fef439bee61144769bfe62dddc830113380624e)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510216 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397658 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339730 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021602036 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)

=== 2026-06-01 ===
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978281 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757969997 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864799 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719863204 (https://github.com/cluebotng/component-configs/commits/9a01604060b738d8bb8b39dd300634a9976b8737)
* 17:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719593861 (https://github.com/cluebotng/component-configs/commits/b7c56032c7bb94a330138a58ef1bf5bb59f8c94c)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577030 (https://github.com/cluebotng/component-configs/commits/9b802544b4b8a568ce3b129be53084ea1f979385)

=== 2026-05-27 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26543153283 (https://github.com/cluebotng/component-configs/commits/b9f54eeee581a4f5a3788d3e16acbc76c7fa6fbb)

=== 2026-05-12 ===
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842151 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562336 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280675 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-08 ===
* 05:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25537718177 (https://github.com/cluebotng/component-configs/commits/ef11c0e9dcb6c448a2eaeb147343007003b2874f)

=== 2026-04-29 ===
* 23:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25139750319 (https://github.com/cluebotng/component-configs/commits/9a101172ba64f47f96b75a6a6d77f65ee589ab4e)

=== 2026-04-23 ===
* 09:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24826851188 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79)
* 05:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24819177834 (https://github.com/cluebotng/component-configs/commits/50ebb073215919775512bff653c7d337c343c315)

=== 2026-04-10 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250823601 (https://github.com/cluebotng/component-configs/commits/78526aec436ada2123387a5fc328bf8e3fe7d4a8)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250714955 (https://github.com/cluebotng/component-configs/commits/6c8100fde23d02e6b289d65ddd7fc06332eabee3)
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442472 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250023341 (https://github.com/cluebotng/component-configs/commits/d9e72fa744a319bc8d37238dc1895ad5d11732ba)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897967 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248770650 (https://github.com/cluebotng/component-configs/commits/30b6b38a296396ea7c907cb624fb55006729e637)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247614901 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247572438 (https://github.com/cluebotng/component-configs/commits/e5facd6bf8968a234c139ac18c8d2e72f8345d9e)

=== 2026-04-09 ===
* 18:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093212 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537335 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-25 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729399 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad)

=== 2026-03-21 ===
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640351 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383554920 (https://github.com/cluebotng/component-configs/commits/87ca816f48bdfbf0cbb10d469dacfe87cefd0184)
* 16:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383551602 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)

=== 2026-02-15 ===
* 10:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380287 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb)

=== 2025-11-11 ===
* 15:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642949 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301618 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4)

=== 2025-11-09 ===
* 22:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215642431 (https://github.com/cluebotng/component-configs/commits/68b314152d3679c2a780d1247682bbceaf08ee20)

=== 2025-11-05 ===
* 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995191 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217950 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101416 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460177 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)
* 09:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092072445 (https://github.com/cluebotng/component-configs/commits/a0d50b624a6cdfa221225a08b11c52ed85e54d0c)

=== 2025-09-26 ===
* 18:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649035 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926539 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d)

=== 2025-09-25 ===
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998408 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801883 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221945 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782309 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641380 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c)

=== 2025-09-24 ===
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198519 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820843 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009170 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983738823 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 19:10 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925753138
* 19:01 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925551560
* 18:50 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925326118

=== 2025-08-29 ===
* 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-15 ===
* 21:12 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 12:55 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:28 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:07 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-11 ===
* 12:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-10 ===
* 17:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-08 ===
* 15:11 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main
* 14:53 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-07 ===
* 15:35 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27

=== 2025-05-22 ===
* 14:42 taavi: cleanup 200G+ of old log files per [[phab:T395006|T395006]]


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>brait0uoklx6vdahsotyt41krvnnu2t</sha1>
    </revision>
    <revision>
      <id>2478993</id>
      <parentid>2478983</parentid>
      <timestamp>2026-11-07T12:39:32Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017605 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)</comment>
      <origin>2478993</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="28977" sha1="f6jch5e7kdnq6s9id1ryyl5zxulp94v" xml:space="preserve">=== 2026-11-07 ===
* 12:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017605 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819817 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250564899 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248723213 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833553 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233691092 (https://github.com/cluebotng/component-configs/commits/b630be8dad4a568f64c5a4b612248b3a15b220a3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248999 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224187997 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485410 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42200307291 (https://github.com/cluebotng/component-configs/commits/255eec1d87ca571dcfd0f09253a9504e7d2b70f1)
* 07:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181945089 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297535 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 04:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42162423773 (https://github.com/cluebotng/component-configs/commits/f0d701fcec7061cd6d2bf7ea127a73895e4a005f)

=== 2026-11-05 ===
* 05:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41976470140 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974543465 (https://github.com/cluebotng/component-configs/commits/7f5f130ad27cc6b58d18dc7db9fb6f17a994795f)
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300445 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214249 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107416 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208797 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274531 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064637 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2)

=== 2026-11-03 ===
* 18:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580691 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616232 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215680 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264114 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696321 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216275 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)

=== 2026-11-01 ===
* 21:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41350442836 (https://github.com/cluebotng/component-configs/commits/9f8d02aa99dd4b425755d1cf842fbce1fa045266)
* 12:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41306290028 (https://github.com/cluebotng/component-configs/commits/d2705aba98618c9fa1aecc8578e6861ed2ef9ef8)

=== 2026-10-31 ===
* 14:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074738 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347156 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603045 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529472 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 10:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191380260 (https://github.com/cluebotng/component-configs/commits/2be620aa8d34f581c9c9bb72401e2bcb82705c7d)

=== 2026-10-30 ===
* 22:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41139892792 (https://github.com/cluebotng/component-configs/commits/ce1c920d2a60015846167ad5fba607bfa99e27fa)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610878 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626139 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:48 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761792 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)

=== 2026-10-29 ===
* 15:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902680991 (https://github.com/cluebotng/component-configs/commits/da88cb5e1e9693171a65562139a9fbf29fa65dfe)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902241168 (https://github.com/cluebotng/component-configs/commits/77ffbc8ebcebe0ba0b96808c0324623bd61aadf9)
* 15:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676116 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757791 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898265897 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)
* 14:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40894578776 (https://github.com/cluebotng/component-configs/commits/d180ba6310a8569d7c091af3c9dd92dcdbda1991)

=== 2026-10-28 ===
* 18:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217679 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe)
* 18:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741508078 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788367 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636878933 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)

=== 2026-10-27 ===
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112336 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 14:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783500 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 13:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108521 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591043 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 08:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249856 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 05:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721145 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518212 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826228 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9)
* 04:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765837 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375498 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527051 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895988 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736579 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6)

=== 2026-10-26 ===
* 23:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679333 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a)

=== 2026-09-01 ===
* 19:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856145 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3)

=== 2026-07-13 ===
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29258212925 (https://github.com/cluebotng/component-configs/commits/2d491fa13ec950460297fa6cfae6801b75d17cf4)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412753 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506690 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804682 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 11:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27615770915 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 14:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179699 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-13 ===
* 13:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27467463967 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285127629 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27280628230 (https://github.com/cluebotng/component-configs/commits/9b8bd8bb539a243486f5bdbbee92001379161762)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788856 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276642792 (https://github.com/cluebotng/component-configs/commits/0fef439bee61144769bfe62dddc830113380624e)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510216 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397658 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339730 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021602036 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)

=== 2026-06-01 ===
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978281 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757969997 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864799 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719863204 (https://github.com/cluebotng/component-configs/commits/9a01604060b738d8bb8b39dd300634a9976b8737)
* 17:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719593861 (https://github.com/cluebotng/component-configs/commits/b7c56032c7bb94a330138a58ef1bf5bb59f8c94c)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577030 (https://github.com/cluebotng/component-configs/commits/9b802544b4b8a568ce3b129be53084ea1f979385)

=== 2026-05-27 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26543153283 (https://github.com/cluebotng/component-configs/commits/b9f54eeee581a4f5a3788d3e16acbc76c7fa6fbb)

=== 2026-05-12 ===
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842151 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562336 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280675 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-08 ===
* 05:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25537718177 (https://github.com/cluebotng/component-configs/commits/ef11c0e9dcb6c448a2eaeb147343007003b2874f)

=== 2026-04-29 ===
* 23:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25139750319 (https://github.com/cluebotng/component-configs/commits/9a101172ba64f47f96b75a6a6d77f65ee589ab4e)

=== 2026-04-23 ===
* 09:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24826851188 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79)
* 05:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24819177834 (https://github.com/cluebotng/component-configs/commits/50ebb073215919775512bff653c7d337c343c315)

=== 2026-04-10 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250823601 (https://github.com/cluebotng/component-configs/commits/78526aec436ada2123387a5fc328bf8e3fe7d4a8)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250714955 (https://github.com/cluebotng/component-configs/commits/6c8100fde23d02e6b289d65ddd7fc06332eabee3)
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442472 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250023341 (https://github.com/cluebotng/component-configs/commits/d9e72fa744a319bc8d37238dc1895ad5d11732ba)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897967 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248770650 (https://github.com/cluebotng/component-configs/commits/30b6b38a296396ea7c907cb624fb55006729e637)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247614901 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247572438 (https://github.com/cluebotng/component-configs/commits/e5facd6bf8968a234c139ac18c8d2e72f8345d9e)

=== 2026-04-09 ===
* 18:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093212 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537335 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-25 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729399 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad)

=== 2026-03-21 ===
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640351 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383554920 (https://github.com/cluebotng/component-configs/commits/87ca816f48bdfbf0cbb10d469dacfe87cefd0184)
* 16:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383551602 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)

=== 2026-02-15 ===
* 10:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380287 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb)

=== 2025-11-11 ===
* 15:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642949 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301618 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4)

=== 2025-11-09 ===
* 22:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215642431 (https://github.com/cluebotng/component-configs/commits/68b314152d3679c2a780d1247682bbceaf08ee20)

=== 2025-11-05 ===
* 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995191 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217950 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101416 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460177 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)
* 09:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092072445 (https://github.com/cluebotng/component-configs/commits/a0d50b624a6cdfa221225a08b11c52ed85e54d0c)

=== 2025-09-26 ===
* 18:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649035 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926539 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d)

=== 2025-09-25 ===
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998408 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801883 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221945 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782309 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641380 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c)

=== 2025-09-24 ===
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198519 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820843 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009170 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983738823 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 19:10 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925753138
* 19:01 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925551560
* 18:50 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925326118

=== 2025-08-29 ===
* 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-15 ===
* 21:12 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 12:55 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:28 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:07 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-11 ===
* 12:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-10 ===
* 17:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-08 ===
* 15:11 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main
* 14:53 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-07 ===
* 15:35 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27

=== 2025-05-22 ===
* 14:42 taavi: cleanup 200G+ of old log files per [[phab:T395006|T395006]]


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>f6jch5e7kdnq6s9id1ryyl5zxulp94v</sha1>
    </revision>
    <revision>
      <id>2478997</id>
      <parentid>2478993</parentid>
      <timestamp>2026-11-07T12:43:37Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373224069 (https://github.com/cluebotng/component-configs/commits/b7987054470c43be35c9e93b1e91e43119048538)</comment>
      <origin>2478997</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="29183" sha1="77ix43nxcjf6g4naf3712wtxk369q7f" xml:space="preserve">=== 2026-11-07 ===
* 12:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373224069 (https://github.com/cluebotng/component-configs/commits/b7987054470c43be35c9e93b1e91e43119048538)
* 12:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017605 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819817 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250564899 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248723213 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833553 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233691092 (https://github.com/cluebotng/component-configs/commits/b630be8dad4a568f64c5a4b612248b3a15b220a3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248999 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224187997 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42216485410 (https://github.com/cluebotng/component-configs/commits/86fca53768dbc4df4fce7786bb9638ebe1a604e6)
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42200307291 (https://github.com/cluebotng/component-configs/commits/255eec1d87ca571dcfd0f09253a9504e7d2b70f1)
* 07:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181945089 (https://github.com/cluebotng/component-configs/commits/da424e45124170000e83bc261fc0ac034e51b840)
* 06:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297535 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 04:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42162423773 (https://github.com/cluebotng/component-configs/commits/f0d701fcec7061cd6d2bf7ea127a73895e4a005f)

=== 2026-11-05 ===
* 05:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41976470140 (https://github.com/cluebotng/component-configs/commits/dfe8bea82a81a0c7ff97c707693c7eff5fc01658)
* 05:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974543465 (https://github.com/cluebotng/component-configs/commits/7f5f130ad27cc6b58d18dc7db9fb6f17a994795f)
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300445 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214249 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36)
* 06:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107416 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208797 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 01:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274531 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6)
* 00:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064637 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2)

=== 2026-11-03 ===
* 18:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693580691 (https://github.com/cluebotng/component-configs/commits/9ff9bae00f7241eef432f91586a0cc47e459c4e7)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616232 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41605215680 (https://github.com/cluebotng/component-configs/commits/874f44a0ab1b450c296132a9246bea96e0b7d25b)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264114 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582696321 (https://github.com/cluebotng/component-configs/commits/085afd71093b712517702f3975ce1a3bea759e1e)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567216275 (https://github.com/cluebotng/component-configs/commits/baa95cf1e3a0b0c85cd8ee23d102608549c93e6d)

=== 2026-11-01 ===
* 21:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41350442836 (https://github.com/cluebotng/component-configs/commits/9f8d02aa99dd4b425755d1cf842fbce1fa045266)
* 12:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41306290028 (https://github.com/cluebotng/component-configs/commits/d2705aba98618c9fa1aecc8578e6861ed2ef9ef8)

=== 2026-10-31 ===
* 14:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41209074738 (https://github.com/cluebotng/component-configs/commits/0bcdc6aa682287740ac2fd88fdeb365a5f53124f)
* 13:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41205347156 (https://github.com/cluebotng/component-configs/commits/9e7894a76b92e294b636c29e6e3f763f01dfd5fb)
* 10:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191603045 (https://github.com/cluebotng/component-configs/commits/0902b192843e7a55f58fe0a50c1494d1f40f11d9)
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529472 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 10:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191380260 (https://github.com/cluebotng/component-configs/commits/2be620aa8d34f581c9c9bb72401e2bcb82705c7d)

=== 2026-10-30 ===
* 22:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41139892792 (https://github.com/cluebotng/component-configs/commits/ce1c920d2a60015846167ad5fba607bfa99e27fa)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610878 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626139 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:48 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125761792 (https://github.com/cluebotng/component-configs/commits/c74bb3ea4a3cb97f66587885106a777f36cd1c1b)

=== 2026-10-29 ===
* 15:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902680991 (https://github.com/cluebotng/component-configs/commits/da88cb5e1e9693171a65562139a9fbf29fa65dfe)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902241168 (https://github.com/cluebotng/component-configs/commits/77ffbc8ebcebe0ba0b96808c0324623bd61aadf9)
* 15:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676116 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757791 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898265897 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)
* 14:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40894578776 (https://github.com/cluebotng/component-configs/commits/d180ba6310a8569d7c091af3c9dd92dcdbda1991)

=== 2026-10-28 ===
* 18:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40748217679 (https://github.com/cluebotng/component-configs/commits/eceec121315d1ae29499bd0543ff1a373edb1efe)
* 18:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40741508078 (https://github.com/cluebotng/component-configs/commits/af850f315b5d274b83e396634fbb4577f83c0c60)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788367 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 05:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40636878933 (https://github.com/cluebotng/component-configs/commits/1519387bf38d5ffaec5380641581f2b9f842af99)

=== 2026-10-27 ===
* 16:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40542112336 (https://github.com/cluebotng/component-configs/commits/e3b42405f7637feca5178261ba1e5662c1009020)
* 14:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40518783500 (https://github.com/cluebotng/component-configs/commits/931b620f8c154ef5e68d899a25f218c171d6de5d)
* 13:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40515108521 (https://github.com/cluebotng/component-configs/commits/e55b6738ff115956fc47c673e403b2d6f91b24a5)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591043 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 08:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40476249856 (https://github.com/cluebotng/component-configs/commits/838abafe65c57870163e3118ce8a093debe0d2e6)
* 05:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455721145 (https://github.com/cluebotng/component-configs/commits/33785ac45923c22c4e14a8b54268e25c98b83f89)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518212 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451826228 (https://github.com/cluebotng/component-configs/commits/264fe43c86ed52f85594cf77c7f12208c886bae9)
* 04:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451765837 (https://github.com/cluebotng/component-configs/commits/b95cee1256fe85e86858e9c35dbf825c29ae24dc)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375498 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527051 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442895988 (https://github.com/cluebotng/component-configs/commits/22ef6144c3a3323d2054a04b4a2fa02c7c9c2cee)
* 03:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40442736579 (https://github.com/cluebotng/component-configs/commits/44d3670909d5f5c7236df807b8156fa2cf0d81e6)

=== 2026-10-26 ===
* 23:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40419679333 (https://github.com/cluebotng/component-configs/commits/c35c9753fc0f5978462a889014ba06ed450c3e0a)

=== 2026-09-01 ===
* 19:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33551856145 (https://github.com/cluebotng/component-configs/commits/0c6ef4655c137d241bf31075ef088ea0dc85c8a3)

=== 2026-07-13 ===
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29258212925 (https://github.com/cluebotng/component-configs/commits/2d491fa13ec950460297fa6cfae6801b75d17cf4)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412753 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506690 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804682 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 11:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27615770915 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 14:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179699 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1)

=== 2026-06-13 ===
* 13:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27467463967 (https://github.com/cluebotng/component-configs/commits/3dc535380a54d2290621b9d585a5018fdc4669a2)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285127629 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27280628230 (https://github.com/cluebotng/component-configs/commits/9b8bd8bb539a243486f5bdbbee92001379161762)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788856 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276642792 (https://github.com/cluebotng/component-configs/commits/0fef439bee61144769bfe62dddc830113380624e)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510216 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275397658 (https://github.com/cluebotng/component-configs/commits/33d203fb0e6b88ac6dc34e82ee630f7d4e6fdb56)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339730 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-05 ===
* 14:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27021602036 (https://github.com/cluebotng/component-configs/commits/d4efd5a504c17f41f2d280dabcb635f9c4f07000)

=== 2026-06-01 ===
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26762978281 (https://github.com/cluebotng/component-configs/commits/9a088c9b8375555c696948825fff7700458b4254)
* 13:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26757969997 (https://github.com/cluebotng/component-configs/commits/4790ebea51ebfbd67e51894987e6273e5940cbf1)

=== 2026-05-31 ===
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864799 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719863204 (https://github.com/cluebotng/component-configs/commits/9a01604060b738d8bb8b39dd300634a9976b8737)
* 17:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719593861 (https://github.com/cluebotng/component-configs/commits/b7c56032c7bb94a330138a58ef1bf5bb59f8c94c)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577030 (https://github.com/cluebotng/component-configs/commits/9b802544b4b8a568ce3b129be53084ea1f979385)

=== 2026-05-27 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26543153283 (https://github.com/cluebotng/component-configs/commits/b9f54eeee581a4f5a3788d3e16acbc76c7fa6fbb)

=== 2026-05-12 ===
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842151 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562336 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280675 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-08 ===
* 05:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25537718177 (https://github.com/cluebotng/component-configs/commits/ef11c0e9dcb6c448a2eaeb147343007003b2874f)

=== 2026-04-29 ===
* 23:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25139750319 (https://github.com/cluebotng/component-configs/commits/9a101172ba64f47f96b75a6a6d77f65ee589ab4e)

=== 2026-04-23 ===
* 09:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24826851188 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79)
* 05:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24819177834 (https://github.com/cluebotng/component-configs/commits/50ebb073215919775512bff653c7d337c343c315)

=== 2026-04-10 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250823601 (https://github.com/cluebotng/component-configs/commits/78526aec436ada2123387a5fc328bf8e3fe7d4a8)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250714955 (https://github.com/cluebotng/component-configs/commits/6c8100fde23d02e6b289d65ddd7fc06332eabee3)
* 15:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442472 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250023341 (https://github.com/cluebotng/component-configs/commits/d9e72fa744a319bc8d37238dc1895ad5d11732ba)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897967 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248770650 (https://github.com/cluebotng/component-configs/commits/30b6b38a296396ea7c907cb624fb55006729e637)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247614901 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247572438 (https://github.com/cluebotng/component-configs/commits/e5facd6bf8968a234c139ac18c8d2e72f8345d9e)

=== 2026-04-09 ===
* 18:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093212 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656791 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537335 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-25 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729399 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad)

=== 2026-03-21 ===
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640351 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383554920 (https://github.com/cluebotng/component-configs/commits/87ca816f48bdfbf0cbb10d469dacfe87cefd0184)
* 16:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383551602 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)

=== 2026-02-15 ===
* 10:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380287 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb)

=== 2025-11-11 ===
* 15:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642949 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301618 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4)

=== 2025-11-09 ===
* 22:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215642431 (https://github.com/cluebotng/component-configs/commits/68b314152d3679c2a780d1247682bbceaf08ee20)

=== 2025-11-05 ===
* 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995191 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217950 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101416 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460177 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)
* 09:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092072445 (https://github.com/cluebotng/component-configs/commits/a0d50b624a6cdfa221225a08b11c52ed85e54d0c)

=== 2025-09-26 ===
* 18:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649035 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926539 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d)

=== 2025-09-25 ===
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998408 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801883 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221945 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782309 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641380 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c)

=== 2025-09-24 ===
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198519 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820843 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009170 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983738823 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 19:10 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925753138
* 19:01 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925551560
* 18:50 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925326118

=== 2025-08-29 ===
* 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-15 ===
* 21:12 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 12:55 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:28 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:07 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-11 ===
* 12:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-10 ===
* 17:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-08 ===
* 15:11 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main
* 14:53 wmbot~damian@tools-bastion-13: report deployed @ refs/heads/main

=== 2025-08-07 ===
* 15:35 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27

=== 2025-05-22 ===
* 14:42 taavi: cleanup 200G+ of old log files per [[phab:T395006|T395006]]


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>77ix43nxcjf6g4naf3712wtxk369q7f</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-monitoring/SAL</title>
    <ns>498</ns>
    <id>459243</id>
    <revision>
      <id>2478973</id>
      <parentid>2478865</parentid>
      <timestamp>2026-11-07T12:29:43Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372350465 (https://github.com/cluebotng/component-configs/commits/48fbc4fed6236580eba63deced350d7a4c952b15)</comment>
      <origin>2478973</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="45883" sha1="qjx56q1iw02m8bher1y7lv5v3q31i78" xml:space="preserve">=== 2026-11-07 ===
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372350465 (https://github.com/cluebotng/component-configs/commits/48fbc4fed6236580eba63deced350d7a4c952b15)

=== 2026-11-06 ===
* 16:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250722611 (https://github.com/cluebotng/component-configs/commits/633d8f1771eebbfc0be6b8168219a96199a12fc9)
* 16:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249872552 (https://github.com/cluebotng/component-configs/commits/76be28c368c96ca51589616db44ce14830dbc7fe)
* 16:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248666990 (https://github.com/cluebotng/component-configs/commits/e4b5844b01d67778506454fa3525d8c2d175c7bb)
* 14:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233881417 (https://github.com/cluebotng/component-configs/commits/13375ca9f2245b8713c6934705cc7b61659a7bf2)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248919 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233207209 (https://github.com/cluebotng/component-configs/commits/4f3156ee11ba46babbca95fca75b3194900de461)
* 13:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42227653941 (https://github.com/cluebotng/component-configs/commits/e9ebfee649f69eaef20695673d0706925b51f232)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188048 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 11:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42210074274 (https://github.com/cluebotng/component-configs/commits/a4f327022b82328ec3d56e77ca3f3f984da6b964)
* 11:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42208073822 (https://github.com/cluebotng/component-configs/commits/6639b38d3900750530ebf6ef8d0a92218849b0f5)
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42200488452 (https://github.com/cluebotng/component-configs/commits/3811ff050c237ab472542a64c46c558d7d159b6d)
* 07:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181039067 (https://github.com/cluebotng/component-configs/commits/6227b943f6ee18f3ac6046651a55ad11e53b6823)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172625760 (https://github.com/cluebotng/component-configs/commits/b4e427ee91fef96236b0b67b43d9e9465d70e2bc)
* 06:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297612 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42171126034 (https://github.com/cluebotng/component-configs/commits/99cc62d4318ecb79d0e5642eab91b3d01e3fe55c)
* 05:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42166166337 (https://github.com/cluebotng/component-configs/commits/533681769ce14b9dc06e88480d711dfd2be43926)
* 05:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42165716583 (https://github.com/cluebotng/component-configs/commits/73465623988d4a1316253b991ecba1de5f9f2902)
* 04:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42160643918 (https://github.com/cluebotng/component-configs/commits/d75eff3c0ab6904c739a0b88046627a0eb019bcc)

=== 2026-11-05 ===
* 06:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41981558450 (https://github.com/cluebotng/component-configs/commits/23de2fc566aa0abfe228312ddd0b1e87b95990eb)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978385836 (https://github.com/cluebotng/component-configs/commits/518d18724ca549296a5ff755fa4e6a576a8c366c)
* 06:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978117359 (https://github.com/cluebotng/component-configs/commits/445aed807c70e37fa8e4e9270743b99bd94a2c3c)
* 06:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41977287826 (https://github.com/cluebotng/component-configs/commits/26aeb742e54365acb8d07e0d79767159175ed5ca)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300597 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 01:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41947942541 (https://github.com/cluebotng/component-configs/commits/fb51e734553eadc617de48a568da790fd164e106)

=== 2026-11-04 ===
* 23:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41930463906 (https://github.com/cluebotng/component-configs/commits/b495272b56a26a89b4498d24e762398799fe3f61)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41878343537 (https://github.com/cluebotng/component-configs/commits/d03a8439ccb347e558a42e94a674edbc6d1b087d)
* 16:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41870962440 (https://github.com/cluebotng/component-configs/commits/3c1b5d1defc502bca33e6c53fe4f68d97f33cfba)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41865536484 (https://github.com/cluebotng/component-configs/commits/358d261c41034263816250714b7bda000b460afa)
* 15:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41861784896 (https://github.com/cluebotng/component-configs/commits/e820717bfa1e2145b54da8fff522fc828da9457f)
* 14:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41850597334 (https://github.com/cluebotng/component-configs/commits/6d4c24a0f2ad8cc6b8220fb6195cac8eca56120e)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41848610075 (https://github.com/cluebotng/component-configs/commits/8c9d33b12cfd03687a781e4cc65df3014ba5ac3f)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208847 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 02:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763724637 (https://github.com/cluebotng/component-configs/commits/1681819283fff40cec7eedcf2e2ddd6b6e555828)
* 02:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763262663 (https://github.com/cluebotng/component-configs/commits/d15c2921f9532c4723d817aa328532a85783a46c)
* 02:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761998436 (https://github.com/cluebotng/component-configs/commits/c16528be19dec3e78a0c5b289774e8fd56d8ab32)
* 02:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761692440 (https://github.com/cluebotng/component-configs/commits/3cdc869413ae6a77faaac98309e3b59dab578a53)
* 02:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41760170800 (https://github.com/cluebotng/component-configs/commits/ee3a8262f4918cfcf2e22ccfe0d728ebf8143df2)

=== 2026-11-03 ===
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41688162297 (https://github.com/cluebotng/component-configs/commits/7f183bdf8d9d62f3092978941b91e04a9a5531c6)
* 12:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616200 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 10:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41622611776 (https://github.com/cluebotng/component-configs/commits/5393f63d09ee5ee78540551dbd2532edbb754bdd)
* 06:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590277348 (https://github.com/cluebotng/component-configs/commits/be302650e64737354887b8a0d3765c836f16937f)
* 06:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590047860 (https://github.com/cluebotng/component-configs/commits/b727998497e3e9aad0bbeead0c3ab4000f61584c)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264099 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582112461 (https://github.com/cluebotng/component-configs/commits/753eb7b9e738029a5443693413aff58e1d3fa349)
* 05:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41581755584 (https://github.com/cluebotng/component-configs/commits/ad0a4ebfac738d2c3b772e79b296d5c00c79c38e)
* 03:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41570372141 (https://github.com/cluebotng/component-configs/commits/9e06eef94c362f2813b0a34a82356b7d503c3f4f)

=== 2026-10-31 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41225627605 (https://github.com/cluebotng/component-configs/commits/3380820548db3fc46a824a5fbc5d33ec7051b5be)
* 16:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41221534882 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 10:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529424 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 10:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191328286 (https://github.com/cluebotng/component-configs/commits/a34d3d663fe29c314bc46372b3918110398eeb21)
* 09:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190383881 (https://github.com/cluebotng/component-configs/commits/129b478f0d25bde6cd074041bd805fef6bf416e4)
* 09:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41190196588 (https://github.com/cluebotng/component-configs/commits/4c53ca15081900bebb4f0a9ad44851060f6b65af)
* 09:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41188223439 (https://github.com/cluebotng/component-configs/commits/f09d1d712110c8ae6993944821fada8410863b84)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41187424593 (https://github.com/cluebotng/component-configs/commits/4e1abde0b1bbe2470301419c6daa5ae306c9b3c9)

=== 2026-10-30 ===
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610802 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626132 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41136968711 (https://github.com/cluebotng/component-configs/commits/9049050317cbf3431cf1cc6413153c4259c7c795)
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41134517923 (https://github.com/cluebotng/component-configs/commits/4692ec54c1fac8de12abbf33a19a479b73fecd8b)
* 21:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41133500657 (https://github.com/cluebotng/component-configs/commits/a10c48e108f64ffe3775152f86d4ef1b2b06bd06)
* 21:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41131191713 (https://github.com/cluebotng/component-configs/commits/99e52c3d1fd981704761f620392d05ab05711dc2)
* 20:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125360788 (https://github.com/cluebotng/component-configs/commits/5ea82c4de6a9b5d0e5c8f6adac04bfacd0752d53)
* 20:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41124106294 (https://github.com/cluebotng/component-configs/commits/fa700f7c00a808236a6b29b814bc843145e1a5aa)

=== 2026-10-29 ===
* 15:38 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902689515 (https://github.com/cluebotng/component-configs/commits/2c61828f93816462b7639fce810e118f8937e7a8)
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902159583 (https://github.com/cluebotng/component-configs/commits/9e1231590a3fc7d063de82c45faf2ab0b4060cdb)
* 15:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900757985 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898265896 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 19:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40755839963 (https://github.com/cluebotng/component-configs/commits/43e1fcf7f4599d40cc9b9bd87abfeba5baede79b)
* 19:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40752366984 (https://github.com/cluebotng/component-configs/commits/b792020c142bd09c08d70ae0f40684c9c3f81079)
* 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40698633627 (https://github.com/cluebotng/component-configs/commits/29d35958a8230b19c67327fae4cc1a2c55f8f4f0)
* 13:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40697046893 (https://github.com/cluebotng/component-configs/commits/65c2635e0eca77c926d6ecfdd3bc4243290cb2de)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788397 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 06:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40643749649 (https://github.com/cluebotng/component-configs/commits/6aa169a5ad67427223a008319e43e00be9db16c5)
* 04:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40632642704 (https://github.com/cluebotng/component-configs/commits/611facf7667c39a7f5b1712e5b174684a0649ea4)

=== 2026-10-27 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40545550695 (https://github.com/cluebotng/component-configs/commits/9a4c6ecd52da46f16ac808838c87f2b4608e2987)
* 16:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40536663681 (https://github.com/cluebotng/component-configs/commits/d02fac71cd9a7c00d0b2cba2a245b95677c46c4d)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40527035651 (https://github.com/cluebotng/component-configs/commits/7dd9e7d26ac0141af54d06f81163a130e5a402d3)
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40526418448 (https://github.com/cluebotng/component-configs/commits/9c1e7dfb720d9ef57fe0e066dcebe9ccd5701eaf)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508590861 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40489900248 (https://github.com/cluebotng/component-configs/commits/8007804f0273db3ecaf618e4e3723dbe1bf0fb67)
* 10:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40487986665 (https://github.com/cluebotng/component-configs/commits/7cc193ac0999f846b516b843914c70a13924f99f)
* 06:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462539944 (https://github.com/cluebotng/component-configs/commits/e4c07d7c9925e5e434298a95070b8f40a7db75c8)
* 06:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459831650 (https://github.com/cluebotng/component-configs/commits/bfbe8dc9171d388710c1d5eface2d00809cd79dd)
* 05:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40456450428 (https://github.com/cluebotng/component-configs/commits/b0fce42989bc739f2992831884538d17c4220a07)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453802690 (https://github.com/cluebotng/component-configs/commits/4186d7d35e3637e5f7722d66c028994026229d84)
* 05:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453565998 (https://github.com/cluebotng/component-configs/commits/0b40c20b317be8b21d7e6b4a9c452f2be1a63558)
* 05:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518130 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451550237 (https://github.com/cluebotng/component-configs/commits/d19590e442b664819343d4373714879e039ae49f)
* 04:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40448492483 (https://github.com/cluebotng/component-configs/commits/d09fba9de2a42159fa1c899076588e5408ccc19b)
* 04:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447571824 (https://github.com/cluebotng/component-configs/commits/7d10e42f436c05ef56f6f317648388d592c3c05f)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375410 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526995 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445124134 (https://github.com/cluebotng/component-configs/commits/f7026cd7409b14b6865323d50c6942fde358921e)

=== 2026-09-01 ===
* 16:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019250 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53)

=== 2026-07-02 ===
* 21:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-20 ===
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27870870297 (https://github.com/cluebotng/component-configs/commits/331545fc215d0e72512d841be6ac751372a6c229)

=== 2026-06-16 ===
* 12:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27618465342 (https://github.com/cluebotng/component-configs/commits/9b3fcbbd479041d764ab53f0a74027e2df6df4f6)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617439342 (https://github.com/cluebotng/component-configs/commits/25d55050b7cd85801d40ba66cf804d6d4a3b51bd)
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617061512 (https://github.com/cluebotng/component-configs/commits/4505fa21f50c25710eaf644eecdc0233f726c946)
* 12:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804749 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 12:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616668922 (https://github.com/cluebotng/component-configs/commits/1bfb0487d9d8b3b799e0d9a30269ca314046feba)
* 12:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615852687 (https://github.com/cluebotng/component-configs/commits/c9746412542e654e3ae7337a570727eb5d7195d6)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27615771202 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27560286658 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27557053203 (https://github.com/cluebotng/component-configs/commits/80c3b871693d13296e1c4640105e4a8cabc9d5a4)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27556781475 (https://github.com/cluebotng/component-configs/commits/81d6af2f5fa912449070fe6ac104761023a9960b)
* 14:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554700598 (https://github.com/cluebotng/component-configs/commits/12298f8c7711b0dbc3ebe3196da055b62b307301)

=== 2026-06-10 ===
* 14:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285126180 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788903 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719865829 (https://github.com/cluebotng/component-configs/commits/ea5cbd0bd54e33719b9e05b40b425784b205838f)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719857015 (https://github.com/cluebotng/component-configs/commits/017ef6a67c2650b45476bc9d5021e7abec570d4f)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719584366 (https://github.com/cluebotng/component-configs/commits/db6b6fb5bb2d48eb49c6ceb67565398f3d58dcac)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577987 (https://github.com/cluebotng/component-configs/commits/9d18fa5f3d4d6f4cdcec776c648bb2d93e2ba652)

=== 2026-05-29 ===
* 02:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26614336971 (https://github.com/cluebotng/component-configs/commits/622f302a735e2a285b7ac4f46e726aaf538f8d93)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26611217389 (https://github.com/cluebotng/component-configs/commits/1c99ebab25e9cbf3bd595f45368b3a5e15489941)

=== 2026-05-28 ===
* 23:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609327945 (https://github.com/cluebotng/component-configs/commits/3772f3cb5ead5bf2f5baf68cb048123a3b98c006)
* 23:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26607745901 (https://github.com/cluebotng/component-configs/commits/4cb42c1d89330943cefe6c3f647fe33e1760d090)
* 21:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26603560881 (https://github.com/cluebotng/component-configs/commits/c78caa8aac55b55f6da890066aba0130382c840f)
* 19:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26598011849 (https://github.com/cluebotng/component-configs/commits/0c29ead3e5562572c27b58851d98362fc1222a01)
* 19:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26595835360 (https://github.com/cluebotng/component-configs/commits/b6254ea98d0e678578264c75442834eb5f9b6421)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26587853908 (https://github.com/cluebotng/component-configs/commits/e6b9e2d0e00d40589e0bebda0bbbffdc178bea3a)
* 11:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26571205214 (https://github.com/cluebotng/component-configs/commits/8f023304162d9582d5736568d9d7078c10b9041d)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26568183905 (https://github.com/cluebotng/component-configs/commits/21b5bddbae9516620318fc278d5c3d1e3f295511)

=== 2026-05-21 ===
* 22:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256956868 (https://github.com/cluebotng/component-configs/commits/95f311a49172258bfd4d1c00d74a918a7aceab68)
* 22:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256861316 (https://github.com/cluebotng/component-configs/commits/e4c68154f62c8846320103abb0e97af63b20f9b1)
* 21:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254813158 (https://github.com/cluebotng/component-configs/commits/ebf46d18f22eee769feeeffd6f4603ec4f320bbb)
* 21:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254787328 (https://github.com/cluebotng/component-configs/commits/80d9748516b3523cecfc76bd85d662fe91dde0c4)

=== 2026-05-19 ===
* 02:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26072649347 (https://github.com/cluebotng/component-configs/commits/94b7da488f5c23df52fa2e750438a3a80f589f3c)
* 02:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071765151 (https://github.com/cluebotng/component-configs/commits/5184d69b34b8d2442df1fbbd652535f5890b4804)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26070598819 (https://github.com/cluebotng/component-configs/commits/c5378a5646858d02f07ababdb2db2955000ab187)

=== 2026-05-18 ===
* 23:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26066535621 (https://github.com/cluebotng/component-configs/commits/2b0b0bbd4e0a727a468343823d4343fe7a082ef9)
* 23:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26065893095 (https://github.com/cluebotng/component-configs/commits/0bf6e1847388fd90f2c03acdc1b80d388cb437e7)

=== 2026-05-17 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986256508 (https://github.com/cluebotng/component-configs/commits/fcc6e35d16a8ccd601ed7f37bbafcc9b26b8f80f)
* 01:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977577391 (https://github.com/cluebotng/component-configs/commits/b7ea6acca0022ec902075282f7fd686162948bc8)
* 01:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>qjx56q1iw02m8bher1y7lv5v3q31i78</sha1>
    </revision>
    <revision>
      <id>2478980</id>
      <parentid>2478973</parentid>
      <timestamp>2026-11-07T12:34:07Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372358825 (https://github.com/cluebotng/component-configs/commits/0f7031a50a4ebd3ca73184235a595780de8e5139)</comment>
      <origin>2478980</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="46089" sha1="hokhrzeavb8j6b0c9f0nbjfsjamsoyp" xml:space="preserve">=== 2026-11-07 ===
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372358825 (https://github.com/cluebotng/component-configs/commits/0f7031a50a4ebd3ca73184235a595780de8e5139)
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372350465 (https://github.com/cluebotng/component-configs/commits/48fbc4fed6236580eba63deced350d7a4c952b15)

=== 2026-11-06 ===
* 16:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250722611 (https://github.com/cluebotng/component-configs/commits/633d8f1771eebbfc0be6b8168219a96199a12fc9)
* 16:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249872552 (https://github.com/cluebotng/component-configs/commits/76be28c368c96ca51589616db44ce14830dbc7fe)
* 16:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248666990 (https://github.com/cluebotng/component-configs/commits/e4b5844b01d67778506454fa3525d8c2d175c7bb)
* 14:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233881417 (https://github.com/cluebotng/component-configs/commits/13375ca9f2245b8713c6934705cc7b61659a7bf2)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248919 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233207209 (https://github.com/cluebotng/component-configs/commits/4f3156ee11ba46babbca95fca75b3194900de461)
* 13:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42227653941 (https://github.com/cluebotng/component-configs/commits/e9ebfee649f69eaef20695673d0706925b51f232)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188048 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 11:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42210074274 (https://github.com/cluebotng/component-configs/commits/a4f327022b82328ec3d56e77ca3f3f984da6b964)
* 11:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42208073822 (https://github.com/cluebotng/component-configs/commits/6639b38d3900750530ebf6ef8d0a92218849b0f5)
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42200488452 (https://github.com/cluebotng/component-configs/commits/3811ff050c237ab472542a64c46c558d7d159b6d)
* 07:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181039067 (https://github.com/cluebotng/component-configs/commits/6227b943f6ee18f3ac6046651a55ad11e53b6823)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172625760 (https://github.com/cluebotng/component-configs/commits/b4e427ee91fef96236b0b67b43d9e9465d70e2bc)
* 06:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297612 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42171126034 (https://github.com/cluebotng/component-configs/commits/99cc62d4318ecb79d0e5642eab91b3d01e3fe55c)
* 05:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42166166337 (https://github.com/cluebotng/component-configs/commits/533681769ce14b9dc06e88480d711dfd2be43926)
* 05:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42165716583 (https://github.com/cluebotng/component-configs/commits/73465623988d4a1316253b991ecba1de5f9f2902)
* 04:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42160643918 (https://github.com/cluebotng/component-configs/commits/d75eff3c0ab6904c739a0b88046627a0eb019bcc)

=== 2026-11-05 ===
* 06:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41981558450 (https://github.com/cluebotng/component-configs/commits/23de2fc566aa0abfe228312ddd0b1e87b95990eb)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978385836 (https://github.com/cluebotng/component-configs/commits/518d18724ca549296a5ff755fa4e6a576a8c366c)
* 06:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978117359 (https://github.com/cluebotng/component-configs/commits/445aed807c70e37fa8e4e9270743b99bd94a2c3c)
* 06:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41977287826 (https://github.com/cluebotng/component-configs/commits/26aeb742e54365acb8d07e0d79767159175ed5ca)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300597 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 01:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41947942541 (https://github.com/cluebotng/component-configs/commits/fb51e734553eadc617de48a568da790fd164e106)

=== 2026-11-04 ===
* 23:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41930463906 (https://github.com/cluebotng/component-configs/commits/b495272b56a26a89b4498d24e762398799fe3f61)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41878343537 (https://github.com/cluebotng/component-configs/commits/d03a8439ccb347e558a42e94a674edbc6d1b087d)
* 16:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41870962440 (https://github.com/cluebotng/component-configs/commits/3c1b5d1defc502bca33e6c53fe4f68d97f33cfba)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41865536484 (https://github.com/cluebotng/component-configs/commits/358d261c41034263816250714b7bda000b460afa)
* 15:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41861784896 (https://github.com/cluebotng/component-configs/commits/e820717bfa1e2145b54da8fff522fc828da9457f)
* 14:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41850597334 (https://github.com/cluebotng/component-configs/commits/6d4c24a0f2ad8cc6b8220fb6195cac8eca56120e)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41848610075 (https://github.com/cluebotng/component-configs/commits/8c9d33b12cfd03687a781e4cc65df3014ba5ac3f)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208847 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 02:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763724637 (https://github.com/cluebotng/component-configs/commits/1681819283fff40cec7eedcf2e2ddd6b6e555828)
* 02:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763262663 (https://github.com/cluebotng/component-configs/commits/d15c2921f9532c4723d817aa328532a85783a46c)
* 02:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761998436 (https://github.com/cluebotng/component-configs/commits/c16528be19dec3e78a0c5b289774e8fd56d8ab32)
* 02:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761692440 (https://github.com/cluebotng/component-configs/commits/3cdc869413ae6a77faaac98309e3b59dab578a53)
* 02:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41760170800 (https://github.com/cluebotng/component-configs/commits/ee3a8262f4918cfcf2e22ccfe0d728ebf8143df2)

=== 2026-11-03 ===
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41688162297 (https://github.com/cluebotng/component-configs/commits/7f183bdf8d9d62f3092978941b91e04a9a5531c6)
* 12:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616200 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 10:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41622611776 (https://github.com/cluebotng/component-configs/commits/5393f63d09ee5ee78540551dbd2532edbb754bdd)
* 06:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590277348 (https://github.com/cluebotng/component-configs/commits/be302650e64737354887b8a0d3765c836f16937f)
* 06:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590047860 (https://github.com/cluebotng/component-configs/commits/b727998497e3e9aad0bbeead0c3ab4000f61584c)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264099 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582112461 (https://github.com/cluebotng/component-configs/commits/753eb7b9e738029a5443693413aff58e1d3fa349)
* 05:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41581755584 (https://github.com/cluebotng/component-configs/commits/ad0a4ebfac738d2c3b772e79b296d5c00c79c38e)
* 03:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41570372141 (https://github.com/cluebotng/component-configs/commits/9e06eef94c362f2813b0a34a82356b7d503c3f4f)

=== 2026-10-31 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41225627605 (https://github.com/cluebotng/component-configs/commits/3380820548db3fc46a824a5fbc5d33ec7051b5be)
* 16:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41221534882 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 10:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529424 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 10:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191328286 (https://github.com/cluebotng/component-configs/commits/a34d3d663fe29c314bc46372b3918110398eeb21)
* 09:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190383881 (https://github.com/cluebotng/component-configs/commits/129b478f0d25bde6cd074041bd805fef6bf416e4)
* 09:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41190196588 (https://github.com/cluebotng/component-configs/commits/4c53ca15081900bebb4f0a9ad44851060f6b65af)
* 09:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41188223439 (https://github.com/cluebotng/component-configs/commits/f09d1d712110c8ae6993944821fada8410863b84)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41187424593 (https://github.com/cluebotng/component-configs/commits/4e1abde0b1bbe2470301419c6daa5ae306c9b3c9)

=== 2026-10-30 ===
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610802 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626132 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41136968711 (https://github.com/cluebotng/component-configs/commits/9049050317cbf3431cf1cc6413153c4259c7c795)
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41134517923 (https://github.com/cluebotng/component-configs/commits/4692ec54c1fac8de12abbf33a19a479b73fecd8b)
* 21:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41133500657 (https://github.com/cluebotng/component-configs/commits/a10c48e108f64ffe3775152f86d4ef1b2b06bd06)
* 21:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41131191713 (https://github.com/cluebotng/component-configs/commits/99e52c3d1fd981704761f620392d05ab05711dc2)
* 20:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125360788 (https://github.com/cluebotng/component-configs/commits/5ea82c4de6a9b5d0e5c8f6adac04bfacd0752d53)
* 20:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41124106294 (https://github.com/cluebotng/component-configs/commits/fa700f7c00a808236a6b29b814bc843145e1a5aa)

=== 2026-10-29 ===
* 15:38 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902689515 (https://github.com/cluebotng/component-configs/commits/2c61828f93816462b7639fce810e118f8937e7a8)
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902159583 (https://github.com/cluebotng/component-configs/commits/9e1231590a3fc7d063de82c45faf2ab0b4060cdb)
* 15:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900757985 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898265896 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 19:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40755839963 (https://github.com/cluebotng/component-configs/commits/43e1fcf7f4599d40cc9b9bd87abfeba5baede79b)
* 19:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40752366984 (https://github.com/cluebotng/component-configs/commits/b792020c142bd09c08d70ae0f40684c9c3f81079)
* 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40698633627 (https://github.com/cluebotng/component-configs/commits/29d35958a8230b19c67327fae4cc1a2c55f8f4f0)
* 13:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40697046893 (https://github.com/cluebotng/component-configs/commits/65c2635e0eca77c926d6ecfdd3bc4243290cb2de)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788397 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 06:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40643749649 (https://github.com/cluebotng/component-configs/commits/6aa169a5ad67427223a008319e43e00be9db16c5)
* 04:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40632642704 (https://github.com/cluebotng/component-configs/commits/611facf7667c39a7f5b1712e5b174684a0649ea4)

=== 2026-10-27 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40545550695 (https://github.com/cluebotng/component-configs/commits/9a4c6ecd52da46f16ac808838c87f2b4608e2987)
* 16:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40536663681 (https://github.com/cluebotng/component-configs/commits/d02fac71cd9a7c00d0b2cba2a245b95677c46c4d)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40527035651 (https://github.com/cluebotng/component-configs/commits/7dd9e7d26ac0141af54d06f81163a130e5a402d3)
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40526418448 (https://github.com/cluebotng/component-configs/commits/9c1e7dfb720d9ef57fe0e066dcebe9ccd5701eaf)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508590861 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40489900248 (https://github.com/cluebotng/component-configs/commits/8007804f0273db3ecaf618e4e3723dbe1bf0fb67)
* 10:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40487986665 (https://github.com/cluebotng/component-configs/commits/7cc193ac0999f846b516b843914c70a13924f99f)
* 06:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462539944 (https://github.com/cluebotng/component-configs/commits/e4c07d7c9925e5e434298a95070b8f40a7db75c8)
* 06:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459831650 (https://github.com/cluebotng/component-configs/commits/bfbe8dc9171d388710c1d5eface2d00809cd79dd)
* 05:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40456450428 (https://github.com/cluebotng/component-configs/commits/b0fce42989bc739f2992831884538d17c4220a07)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453802690 (https://github.com/cluebotng/component-configs/commits/4186d7d35e3637e5f7722d66c028994026229d84)
* 05:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453565998 (https://github.com/cluebotng/component-configs/commits/0b40c20b317be8b21d7e6b4a9c452f2be1a63558)
* 05:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518130 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451550237 (https://github.com/cluebotng/component-configs/commits/d19590e442b664819343d4373714879e039ae49f)
* 04:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40448492483 (https://github.com/cluebotng/component-configs/commits/d09fba9de2a42159fa1c899076588e5408ccc19b)
* 04:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447571824 (https://github.com/cluebotng/component-configs/commits/7d10e42f436c05ef56f6f317648388d592c3c05f)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375410 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526995 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445124134 (https://github.com/cluebotng/component-configs/commits/f7026cd7409b14b6865323d50c6942fde358921e)

=== 2026-09-01 ===
* 16:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019250 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53)

=== 2026-07-02 ===
* 21:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-20 ===
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27870870297 (https://github.com/cluebotng/component-configs/commits/331545fc215d0e72512d841be6ac751372a6c229)

=== 2026-06-16 ===
* 12:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27618465342 (https://github.com/cluebotng/component-configs/commits/9b3fcbbd479041d764ab53f0a74027e2df6df4f6)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617439342 (https://github.com/cluebotng/component-configs/commits/25d55050b7cd85801d40ba66cf804d6d4a3b51bd)
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617061512 (https://github.com/cluebotng/component-configs/commits/4505fa21f50c25710eaf644eecdc0233f726c946)
* 12:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804749 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 12:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616668922 (https://github.com/cluebotng/component-configs/commits/1bfb0487d9d8b3b799e0d9a30269ca314046feba)
* 12:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615852687 (https://github.com/cluebotng/component-configs/commits/c9746412542e654e3ae7337a570727eb5d7195d6)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27615771202 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27560286658 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27557053203 (https://github.com/cluebotng/component-configs/commits/80c3b871693d13296e1c4640105e4a8cabc9d5a4)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27556781475 (https://github.com/cluebotng/component-configs/commits/81d6af2f5fa912449070fe6ac104761023a9960b)
* 14:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554700598 (https://github.com/cluebotng/component-configs/commits/12298f8c7711b0dbc3ebe3196da055b62b307301)

=== 2026-06-10 ===
* 14:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285126180 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788903 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719865829 (https://github.com/cluebotng/component-configs/commits/ea5cbd0bd54e33719b9e05b40b425784b205838f)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719857015 (https://github.com/cluebotng/component-configs/commits/017ef6a67c2650b45476bc9d5021e7abec570d4f)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719584366 (https://github.com/cluebotng/component-configs/commits/db6b6fb5bb2d48eb49c6ceb67565398f3d58dcac)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577987 (https://github.com/cluebotng/component-configs/commits/9d18fa5f3d4d6f4cdcec776c648bb2d93e2ba652)

=== 2026-05-29 ===
* 02:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26614336971 (https://github.com/cluebotng/component-configs/commits/622f302a735e2a285b7ac4f46e726aaf538f8d93)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26611217389 (https://github.com/cluebotng/component-configs/commits/1c99ebab25e9cbf3bd595f45368b3a5e15489941)

=== 2026-05-28 ===
* 23:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609327945 (https://github.com/cluebotng/component-configs/commits/3772f3cb5ead5bf2f5baf68cb048123a3b98c006)
* 23:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26607745901 (https://github.com/cluebotng/component-configs/commits/4cb42c1d89330943cefe6c3f647fe33e1760d090)
* 21:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26603560881 (https://github.com/cluebotng/component-configs/commits/c78caa8aac55b55f6da890066aba0130382c840f)
* 19:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26598011849 (https://github.com/cluebotng/component-configs/commits/0c29ead3e5562572c27b58851d98362fc1222a01)
* 19:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26595835360 (https://github.com/cluebotng/component-configs/commits/b6254ea98d0e678578264c75442834eb5f9b6421)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26587853908 (https://github.com/cluebotng/component-configs/commits/e6b9e2d0e00d40589e0bebda0bbbffdc178bea3a)
* 11:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26571205214 (https://github.com/cluebotng/component-configs/commits/8f023304162d9582d5736568d9d7078c10b9041d)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26568183905 (https://github.com/cluebotng/component-configs/commits/21b5bddbae9516620318fc278d5c3d1e3f295511)

=== 2026-05-21 ===
* 22:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256956868 (https://github.com/cluebotng/component-configs/commits/95f311a49172258bfd4d1c00d74a918a7aceab68)
* 22:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256861316 (https://github.com/cluebotng/component-configs/commits/e4c68154f62c8846320103abb0e97af63b20f9b1)
* 21:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254813158 (https://github.com/cluebotng/component-configs/commits/ebf46d18f22eee769feeeffd6f4603ec4f320bbb)
* 21:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254787328 (https://github.com/cluebotng/component-configs/commits/80d9748516b3523cecfc76bd85d662fe91dde0c4)

=== 2026-05-19 ===
* 02:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26072649347 (https://github.com/cluebotng/component-configs/commits/94b7da488f5c23df52fa2e750438a3a80f589f3c)
* 02:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071765151 (https://github.com/cluebotng/component-configs/commits/5184d69b34b8d2442df1fbbd652535f5890b4804)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26070598819 (https://github.com/cluebotng/component-configs/commits/c5378a5646858d02f07ababdb2db2955000ab187)

=== 2026-05-18 ===
* 23:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26066535621 (https://github.com/cluebotng/component-configs/commits/2b0b0bbd4e0a727a468343823d4343fe7a082ef9)
* 23:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26065893095 (https://github.com/cluebotng/component-configs/commits/0bf6e1847388fd90f2c03acdc1b80d388cb437e7)

=== 2026-05-17 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986256508 (https://github.com/cluebotng/component-configs/commits/fcc6e35d16a8ccd601ed7f37bbafcc9b26b8f80f)
* 01:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977577391 (https://github.com/cluebotng/component-configs/commits/b7ea6acca0022ec902075282f7fd686162948bc8)
* 01:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>hokhrzeavb8j6b0c9f0nbjfsjamsoyp</sha1>
    </revision>
    <revision>
      <id>2478994</id>
      <parentid>2478980</parentid>
      <timestamp>2026-11-07T12:39:36Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373062046 (https://github.com/cluebotng/component-configs/commits/acb2a23da058868d4d36155c4ddac250479dadc2)</comment>
      <origin>2478994</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="46295" sha1="5qt4e2yroh85fx6j9mpj7shjtqx4nqa" xml:space="preserve">=== 2026-11-07 ===
* 12:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373062046 (https://github.com/cluebotng/component-configs/commits/acb2a23da058868d4d36155c4ddac250479dadc2)
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372358825 (https://github.com/cluebotng/component-configs/commits/0f7031a50a4ebd3ca73184235a595780de8e5139)
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372350465 (https://github.com/cluebotng/component-configs/commits/48fbc4fed6236580eba63deced350d7a4c952b15)

=== 2026-11-06 ===
* 16:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250722611 (https://github.com/cluebotng/component-configs/commits/633d8f1771eebbfc0be6b8168219a96199a12fc9)
* 16:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249872552 (https://github.com/cluebotng/component-configs/commits/76be28c368c96ca51589616db44ce14830dbc7fe)
* 16:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248666990 (https://github.com/cluebotng/component-configs/commits/e4b5844b01d67778506454fa3525d8c2d175c7bb)
* 14:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233881417 (https://github.com/cluebotng/component-configs/commits/13375ca9f2245b8713c6934705cc7b61659a7bf2)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248919 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233207209 (https://github.com/cluebotng/component-configs/commits/4f3156ee11ba46babbca95fca75b3194900de461)
* 13:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42227653941 (https://github.com/cluebotng/component-configs/commits/e9ebfee649f69eaef20695673d0706925b51f232)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188048 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 11:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42210074274 (https://github.com/cluebotng/component-configs/commits/a4f327022b82328ec3d56e77ca3f3f984da6b964)
* 11:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42208073822 (https://github.com/cluebotng/component-configs/commits/6639b38d3900750530ebf6ef8d0a92218849b0f5)
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42200488452 (https://github.com/cluebotng/component-configs/commits/3811ff050c237ab472542a64c46c558d7d159b6d)
* 07:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181039067 (https://github.com/cluebotng/component-configs/commits/6227b943f6ee18f3ac6046651a55ad11e53b6823)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172625760 (https://github.com/cluebotng/component-configs/commits/b4e427ee91fef96236b0b67b43d9e9465d70e2bc)
* 06:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297612 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42171126034 (https://github.com/cluebotng/component-configs/commits/99cc62d4318ecb79d0e5642eab91b3d01e3fe55c)
* 05:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42166166337 (https://github.com/cluebotng/component-configs/commits/533681769ce14b9dc06e88480d711dfd2be43926)
* 05:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42165716583 (https://github.com/cluebotng/component-configs/commits/73465623988d4a1316253b991ecba1de5f9f2902)
* 04:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42160643918 (https://github.com/cluebotng/component-configs/commits/d75eff3c0ab6904c739a0b88046627a0eb019bcc)

=== 2026-11-05 ===
* 06:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41981558450 (https://github.com/cluebotng/component-configs/commits/23de2fc566aa0abfe228312ddd0b1e87b95990eb)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978385836 (https://github.com/cluebotng/component-configs/commits/518d18724ca549296a5ff755fa4e6a576a8c366c)
* 06:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978117359 (https://github.com/cluebotng/component-configs/commits/445aed807c70e37fa8e4e9270743b99bd94a2c3c)
* 06:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41977287826 (https://github.com/cluebotng/component-configs/commits/26aeb742e54365acb8d07e0d79767159175ed5ca)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300597 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 01:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41947942541 (https://github.com/cluebotng/component-configs/commits/fb51e734553eadc617de48a568da790fd164e106)

=== 2026-11-04 ===
* 23:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41930463906 (https://github.com/cluebotng/component-configs/commits/b495272b56a26a89b4498d24e762398799fe3f61)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41878343537 (https://github.com/cluebotng/component-configs/commits/d03a8439ccb347e558a42e94a674edbc6d1b087d)
* 16:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41870962440 (https://github.com/cluebotng/component-configs/commits/3c1b5d1defc502bca33e6c53fe4f68d97f33cfba)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41865536484 (https://github.com/cluebotng/component-configs/commits/358d261c41034263816250714b7bda000b460afa)
* 15:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41861784896 (https://github.com/cluebotng/component-configs/commits/e820717bfa1e2145b54da8fff522fc828da9457f)
* 14:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41850597334 (https://github.com/cluebotng/component-configs/commits/6d4c24a0f2ad8cc6b8220fb6195cac8eca56120e)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41848610075 (https://github.com/cluebotng/component-configs/commits/8c9d33b12cfd03687a781e4cc65df3014ba5ac3f)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208847 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 02:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763724637 (https://github.com/cluebotng/component-configs/commits/1681819283fff40cec7eedcf2e2ddd6b6e555828)
* 02:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763262663 (https://github.com/cluebotng/component-configs/commits/d15c2921f9532c4723d817aa328532a85783a46c)
* 02:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761998436 (https://github.com/cluebotng/component-configs/commits/c16528be19dec3e78a0c5b289774e8fd56d8ab32)
* 02:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761692440 (https://github.com/cluebotng/component-configs/commits/3cdc869413ae6a77faaac98309e3b59dab578a53)
* 02:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41760170800 (https://github.com/cluebotng/component-configs/commits/ee3a8262f4918cfcf2e22ccfe0d728ebf8143df2)

=== 2026-11-03 ===
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41688162297 (https://github.com/cluebotng/component-configs/commits/7f183bdf8d9d62f3092978941b91e04a9a5531c6)
* 12:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616200 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 10:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41622611776 (https://github.com/cluebotng/component-configs/commits/5393f63d09ee5ee78540551dbd2532edbb754bdd)
* 06:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590277348 (https://github.com/cluebotng/component-configs/commits/be302650e64737354887b8a0d3765c836f16937f)
* 06:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590047860 (https://github.com/cluebotng/component-configs/commits/b727998497e3e9aad0bbeead0c3ab4000f61584c)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264099 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582112461 (https://github.com/cluebotng/component-configs/commits/753eb7b9e738029a5443693413aff58e1d3fa349)
* 05:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41581755584 (https://github.com/cluebotng/component-configs/commits/ad0a4ebfac738d2c3b772e79b296d5c00c79c38e)
* 03:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41570372141 (https://github.com/cluebotng/component-configs/commits/9e06eef94c362f2813b0a34a82356b7d503c3f4f)

=== 2026-10-31 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41225627605 (https://github.com/cluebotng/component-configs/commits/3380820548db3fc46a824a5fbc5d33ec7051b5be)
* 16:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41221534882 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 10:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529424 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 10:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191328286 (https://github.com/cluebotng/component-configs/commits/a34d3d663fe29c314bc46372b3918110398eeb21)
* 09:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190383881 (https://github.com/cluebotng/component-configs/commits/129b478f0d25bde6cd074041bd805fef6bf416e4)
* 09:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41190196588 (https://github.com/cluebotng/component-configs/commits/4c53ca15081900bebb4f0a9ad44851060f6b65af)
* 09:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41188223439 (https://github.com/cluebotng/component-configs/commits/f09d1d712110c8ae6993944821fada8410863b84)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41187424593 (https://github.com/cluebotng/component-configs/commits/4e1abde0b1bbe2470301419c6daa5ae306c9b3c9)

=== 2026-10-30 ===
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610802 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626132 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41136968711 (https://github.com/cluebotng/component-configs/commits/9049050317cbf3431cf1cc6413153c4259c7c795)
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41134517923 (https://github.com/cluebotng/component-configs/commits/4692ec54c1fac8de12abbf33a19a479b73fecd8b)
* 21:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41133500657 (https://github.com/cluebotng/component-configs/commits/a10c48e108f64ffe3775152f86d4ef1b2b06bd06)
* 21:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41131191713 (https://github.com/cluebotng/component-configs/commits/99e52c3d1fd981704761f620392d05ab05711dc2)
* 20:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125360788 (https://github.com/cluebotng/component-configs/commits/5ea82c4de6a9b5d0e5c8f6adac04bfacd0752d53)
* 20:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41124106294 (https://github.com/cluebotng/component-configs/commits/fa700f7c00a808236a6b29b814bc843145e1a5aa)

=== 2026-10-29 ===
* 15:38 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902689515 (https://github.com/cluebotng/component-configs/commits/2c61828f93816462b7639fce810e118f8937e7a8)
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902159583 (https://github.com/cluebotng/component-configs/commits/9e1231590a3fc7d063de82c45faf2ab0b4060cdb)
* 15:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900757985 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898265896 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 19:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40755839963 (https://github.com/cluebotng/component-configs/commits/43e1fcf7f4599d40cc9b9bd87abfeba5baede79b)
* 19:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40752366984 (https://github.com/cluebotng/component-configs/commits/b792020c142bd09c08d70ae0f40684c9c3f81079)
* 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40698633627 (https://github.com/cluebotng/component-configs/commits/29d35958a8230b19c67327fae4cc1a2c55f8f4f0)
* 13:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40697046893 (https://github.com/cluebotng/component-configs/commits/65c2635e0eca77c926d6ecfdd3bc4243290cb2de)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788397 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 06:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40643749649 (https://github.com/cluebotng/component-configs/commits/6aa169a5ad67427223a008319e43e00be9db16c5)
* 04:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40632642704 (https://github.com/cluebotng/component-configs/commits/611facf7667c39a7f5b1712e5b174684a0649ea4)

=== 2026-10-27 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40545550695 (https://github.com/cluebotng/component-configs/commits/9a4c6ecd52da46f16ac808838c87f2b4608e2987)
* 16:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40536663681 (https://github.com/cluebotng/component-configs/commits/d02fac71cd9a7c00d0b2cba2a245b95677c46c4d)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40527035651 (https://github.com/cluebotng/component-configs/commits/7dd9e7d26ac0141af54d06f81163a130e5a402d3)
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40526418448 (https://github.com/cluebotng/component-configs/commits/9c1e7dfb720d9ef57fe0e066dcebe9ccd5701eaf)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508590861 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40489900248 (https://github.com/cluebotng/component-configs/commits/8007804f0273db3ecaf618e4e3723dbe1bf0fb67)
* 10:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40487986665 (https://github.com/cluebotng/component-configs/commits/7cc193ac0999f846b516b843914c70a13924f99f)
* 06:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462539944 (https://github.com/cluebotng/component-configs/commits/e4c07d7c9925e5e434298a95070b8f40a7db75c8)
* 06:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459831650 (https://github.com/cluebotng/component-configs/commits/bfbe8dc9171d388710c1d5eface2d00809cd79dd)
* 05:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40456450428 (https://github.com/cluebotng/component-configs/commits/b0fce42989bc739f2992831884538d17c4220a07)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453802690 (https://github.com/cluebotng/component-configs/commits/4186d7d35e3637e5f7722d66c028994026229d84)
* 05:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453565998 (https://github.com/cluebotng/component-configs/commits/0b40c20b317be8b21d7e6b4a9c452f2be1a63558)
* 05:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518130 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451550237 (https://github.com/cluebotng/component-configs/commits/d19590e442b664819343d4373714879e039ae49f)
* 04:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40448492483 (https://github.com/cluebotng/component-configs/commits/d09fba9de2a42159fa1c899076588e5408ccc19b)
* 04:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447571824 (https://github.com/cluebotng/component-configs/commits/7d10e42f436c05ef56f6f317648388d592c3c05f)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375410 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526995 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445124134 (https://github.com/cluebotng/component-configs/commits/f7026cd7409b14b6865323d50c6942fde358921e)

=== 2026-09-01 ===
* 16:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019250 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53)

=== 2026-07-02 ===
* 21:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-20 ===
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27870870297 (https://github.com/cluebotng/component-configs/commits/331545fc215d0e72512d841be6ac751372a6c229)

=== 2026-06-16 ===
* 12:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27618465342 (https://github.com/cluebotng/component-configs/commits/9b3fcbbd479041d764ab53f0a74027e2df6df4f6)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617439342 (https://github.com/cluebotng/component-configs/commits/25d55050b7cd85801d40ba66cf804d6d4a3b51bd)
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617061512 (https://github.com/cluebotng/component-configs/commits/4505fa21f50c25710eaf644eecdc0233f726c946)
* 12:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804749 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 12:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616668922 (https://github.com/cluebotng/component-configs/commits/1bfb0487d9d8b3b799e0d9a30269ca314046feba)
* 12:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615852687 (https://github.com/cluebotng/component-configs/commits/c9746412542e654e3ae7337a570727eb5d7195d6)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27615771202 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27560286658 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27557053203 (https://github.com/cluebotng/component-configs/commits/80c3b871693d13296e1c4640105e4a8cabc9d5a4)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27556781475 (https://github.com/cluebotng/component-configs/commits/81d6af2f5fa912449070fe6ac104761023a9960b)
* 14:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554700598 (https://github.com/cluebotng/component-configs/commits/12298f8c7711b0dbc3ebe3196da055b62b307301)

=== 2026-06-10 ===
* 14:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285126180 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788903 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719865829 (https://github.com/cluebotng/component-configs/commits/ea5cbd0bd54e33719b9e05b40b425784b205838f)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719857015 (https://github.com/cluebotng/component-configs/commits/017ef6a67c2650b45476bc9d5021e7abec570d4f)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719584366 (https://github.com/cluebotng/component-configs/commits/db6b6fb5bb2d48eb49c6ceb67565398f3d58dcac)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577987 (https://github.com/cluebotng/component-configs/commits/9d18fa5f3d4d6f4cdcec776c648bb2d93e2ba652)

=== 2026-05-29 ===
* 02:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26614336971 (https://github.com/cluebotng/component-configs/commits/622f302a735e2a285b7ac4f46e726aaf538f8d93)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26611217389 (https://github.com/cluebotng/component-configs/commits/1c99ebab25e9cbf3bd595f45368b3a5e15489941)

=== 2026-05-28 ===
* 23:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609327945 (https://github.com/cluebotng/component-configs/commits/3772f3cb5ead5bf2f5baf68cb048123a3b98c006)
* 23:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26607745901 (https://github.com/cluebotng/component-configs/commits/4cb42c1d89330943cefe6c3f647fe33e1760d090)
* 21:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26603560881 (https://github.com/cluebotng/component-configs/commits/c78caa8aac55b55f6da890066aba0130382c840f)
* 19:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26598011849 (https://github.com/cluebotng/component-configs/commits/0c29ead3e5562572c27b58851d98362fc1222a01)
* 19:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26595835360 (https://github.com/cluebotng/component-configs/commits/b6254ea98d0e678578264c75442834eb5f9b6421)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26587853908 (https://github.com/cluebotng/component-configs/commits/e6b9e2d0e00d40589e0bebda0bbbffdc178bea3a)
* 11:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26571205214 (https://github.com/cluebotng/component-configs/commits/8f023304162d9582d5736568d9d7078c10b9041d)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26568183905 (https://github.com/cluebotng/component-configs/commits/21b5bddbae9516620318fc278d5c3d1e3f295511)

=== 2026-05-21 ===
* 22:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256956868 (https://github.com/cluebotng/component-configs/commits/95f311a49172258bfd4d1c00d74a918a7aceab68)
* 22:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256861316 (https://github.com/cluebotng/component-configs/commits/e4c68154f62c8846320103abb0e97af63b20f9b1)
* 21:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254813158 (https://github.com/cluebotng/component-configs/commits/ebf46d18f22eee769feeeffd6f4603ec4f320bbb)
* 21:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254787328 (https://github.com/cluebotng/component-configs/commits/80d9748516b3523cecfc76bd85d662fe91dde0c4)

=== 2026-05-19 ===
* 02:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26072649347 (https://github.com/cluebotng/component-configs/commits/94b7da488f5c23df52fa2e750438a3a80f589f3c)
* 02:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071765151 (https://github.com/cluebotng/component-configs/commits/5184d69b34b8d2442df1fbbd652535f5890b4804)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26070598819 (https://github.com/cluebotng/component-configs/commits/c5378a5646858d02f07ababdb2db2955000ab187)

=== 2026-05-18 ===
* 23:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26066535621 (https://github.com/cluebotng/component-configs/commits/2b0b0bbd4e0a727a468343823d4343fe7a082ef9)
* 23:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26065893095 (https://github.com/cluebotng/component-configs/commits/0bf6e1847388fd90f2c03acdc1b80d388cb437e7)

=== 2026-05-17 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986256508 (https://github.com/cluebotng/component-configs/commits/fcc6e35d16a8ccd601ed7f37bbafcc9b26b8f80f)
* 01:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977577391 (https://github.com/cluebotng/component-configs/commits/b7ea6acca0022ec902075282f7fd686162948bc8)
* 01:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>5qt4e2yroh85fx6j9mpj7shjtqx4nqa</sha1>
    </revision>
    <revision>
      <id>2479001</id>
      <parentid>2478994</parentid>
      <timestamp>2026-11-07T12:45:20Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373298737 (https://github.com/cluebotng/component-configs/commits/5583d26dcec45d54de2c90673628b22357c4abc5)</comment>
      <origin>2479001</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="46501" sha1="5iitvv81c65cluy1vufasvj3wh5tu0m" xml:space="preserve">=== 2026-11-07 ===
* 12:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373298737 (https://github.com/cluebotng/component-configs/commits/5583d26dcec45d54de2c90673628b22357c4abc5)
* 12:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373062046 (https://github.com/cluebotng/component-configs/commits/acb2a23da058868d4d36155c4ddac250479dadc2)
* 12:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372358825 (https://github.com/cluebotng/component-configs/commits/0f7031a50a4ebd3ca73184235a595780de8e5139)
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372350465 (https://github.com/cluebotng/component-configs/commits/48fbc4fed6236580eba63deced350d7a4c952b15)

=== 2026-11-06 ===
* 16:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250722611 (https://github.com/cluebotng/component-configs/commits/633d8f1771eebbfc0be6b8168219a96199a12fc9)
* 16:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249872552 (https://github.com/cluebotng/component-configs/commits/76be28c368c96ca51589616db44ce14830dbc7fe)
* 16:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42248666990 (https://github.com/cluebotng/component-configs/commits/e4b5844b01d67778506454fa3525d8c2d175c7bb)
* 14:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233881417 (https://github.com/cluebotng/component-configs/commits/13375ca9f2245b8713c6934705cc7b61659a7bf2)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248919 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233207209 (https://github.com/cluebotng/component-configs/commits/4f3156ee11ba46babbca95fca75b3194900de461)
* 13:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42227653941 (https://github.com/cluebotng/component-configs/commits/e9ebfee649f69eaef20695673d0706925b51f232)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188048 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 11:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42210074274 (https://github.com/cluebotng/component-configs/commits/a4f327022b82328ec3d56e77ca3f3f984da6b964)
* 11:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42208073822 (https://github.com/cluebotng/component-configs/commits/6639b38d3900750530ebf6ef8d0a92218849b0f5)
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42200488452 (https://github.com/cluebotng/component-configs/commits/3811ff050c237ab472542a64c46c558d7d159b6d)
* 07:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42181039067 (https://github.com/cluebotng/component-configs/commits/6227b943f6ee18f3ac6046651a55ad11e53b6823)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172625760 (https://github.com/cluebotng/component-configs/commits/b4e427ee91fef96236b0b67b43d9e9465d70e2bc)
* 06:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297612 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 05:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42171126034 (https://github.com/cluebotng/component-configs/commits/99cc62d4318ecb79d0e5642eab91b3d01e3fe55c)
* 05:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42166166337 (https://github.com/cluebotng/component-configs/commits/533681769ce14b9dc06e88480d711dfd2be43926)
* 05:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42165716583 (https://github.com/cluebotng/component-configs/commits/73465623988d4a1316253b991ecba1de5f9f2902)
* 04:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42160643918 (https://github.com/cluebotng/component-configs/commits/d75eff3c0ab6904c739a0b88046627a0eb019bcc)

=== 2026-11-05 ===
* 06:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41981558450 (https://github.com/cluebotng/component-configs/commits/23de2fc566aa0abfe228312ddd0b1e87b95990eb)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978385836 (https://github.com/cluebotng/component-configs/commits/518d18724ca549296a5ff755fa4e6a576a8c366c)
* 06:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41978117359 (https://github.com/cluebotng/component-configs/commits/445aed807c70e37fa8e4e9270743b99bd94a2c3c)
* 06:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41977287826 (https://github.com/cluebotng/component-configs/commits/26aeb742e54365acb8d07e0d79767159175ed5ca)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300597 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 01:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41947942541 (https://github.com/cluebotng/component-configs/commits/fb51e734553eadc617de48a568da790fd164e106)

=== 2026-11-04 ===
* 23:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41930463906 (https://github.com/cluebotng/component-configs/commits/b495272b56a26a89b4498d24e762398799fe3f61)
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41878343537 (https://github.com/cluebotng/component-configs/commits/d03a8439ccb347e558a42e94a674edbc6d1b087d)
* 16:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41870962440 (https://github.com/cluebotng/component-configs/commits/3c1b5d1defc502bca33e6c53fe4f68d97f33cfba)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41865536484 (https://github.com/cluebotng/component-configs/commits/358d261c41034263816250714b7bda000b460afa)
* 15:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41861784896 (https://github.com/cluebotng/component-configs/commits/e820717bfa1e2145b54da8fff522fc828da9457f)
* 14:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41850597334 (https://github.com/cluebotng/component-configs/commits/6d4c24a0f2ad8cc6b8220fb6195cac8eca56120e)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41848610075 (https://github.com/cluebotng/component-configs/commits/8c9d33b12cfd03687a781e4cc65df3014ba5ac3f)
* 03:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208847 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 02:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763724637 (https://github.com/cluebotng/component-configs/commits/1681819283fff40cec7eedcf2e2ddd6b6e555828)
* 02:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763262663 (https://github.com/cluebotng/component-configs/commits/d15c2921f9532c4723d817aa328532a85783a46c)
* 02:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761998436 (https://github.com/cluebotng/component-configs/commits/c16528be19dec3e78a0c5b289774e8fd56d8ab32)
* 02:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761692440 (https://github.com/cluebotng/component-configs/commits/3cdc869413ae6a77faaac98309e3b59dab578a53)
* 02:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41760170800 (https://github.com/cluebotng/component-configs/commits/ee3a8262f4918cfcf2e22ccfe0d728ebf8143df2)

=== 2026-11-03 ===
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41688162297 (https://github.com/cluebotng/component-configs/commits/7f183bdf8d9d62f3092978941b91e04a9a5531c6)
* 12:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616200 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 10:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41622611776 (https://github.com/cluebotng/component-configs/commits/5393f63d09ee5ee78540551dbd2532edbb754bdd)
* 06:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590277348 (https://github.com/cluebotng/component-configs/commits/be302650e64737354887b8a0d3765c836f16937f)
* 06:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41590047860 (https://github.com/cluebotng/component-configs/commits/b727998497e3e9aad0bbeead0c3ab4000f61584c)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264099 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 05:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41582112461 (https://github.com/cluebotng/component-configs/commits/753eb7b9e738029a5443693413aff58e1d3fa349)
* 05:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41581755584 (https://github.com/cluebotng/component-configs/commits/ad0a4ebfac738d2c3b772e79b296d5c00c79c38e)
* 03:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41570372141 (https://github.com/cluebotng/component-configs/commits/9e06eef94c362f2813b0a34a82356b7d503c3f4f)

=== 2026-10-31 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41225627605 (https://github.com/cluebotng/component-configs/commits/3380820548db3fc46a824a5fbc5d33ec7051b5be)
* 16:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41221534882 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 10:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529424 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 10:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191328286 (https://github.com/cluebotng/component-configs/commits/a34d3d663fe29c314bc46372b3918110398eeb21)
* 09:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190383881 (https://github.com/cluebotng/component-configs/commits/129b478f0d25bde6cd074041bd805fef6bf416e4)
* 09:52 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41190196588 (https://github.com/cluebotng/component-configs/commits/4c53ca15081900bebb4f0a9ad44851060f6b65af)
* 09:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41188223439 (https://github.com/cluebotng/component-configs/commits/f09d1d712110c8ae6993944821fada8410863b84)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41187424593 (https://github.com/cluebotng/component-configs/commits/4e1abde0b1bbe2470301419c6daa5ae306c9b3c9)

=== 2026-10-30 ===
* 22:35 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610802 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626132 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41136968711 (https://github.com/cluebotng/component-configs/commits/9049050317cbf3431cf1cc6413153c4259c7c795)
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41134517923 (https://github.com/cluebotng/component-configs/commits/4692ec54c1fac8de12abbf33a19a479b73fecd8b)
* 21:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41133500657 (https://github.com/cluebotng/component-configs/commits/a10c48e108f64ffe3775152f86d4ef1b2b06bd06)
* 21:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41131191713 (https://github.com/cluebotng/component-configs/commits/99e52c3d1fd981704761f620392d05ab05711dc2)
* 20:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125360788 (https://github.com/cluebotng/component-configs/commits/5ea82c4de6a9b5d0e5c8f6adac04bfacd0752d53)
* 20:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41124106294 (https://github.com/cluebotng/component-configs/commits/fa700f7c00a808236a6b29b814bc843145e1a5aa)

=== 2026-10-29 ===
* 15:38 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902689515 (https://github.com/cluebotng/component-configs/commits/2c61828f93816462b7639fce810e118f8937e7a8)
* 15:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902159583 (https://github.com/cluebotng/component-configs/commits/9e1231590a3fc7d063de82c45faf2ab0b4060cdb)
* 15:29 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40900757985 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898265896 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 19:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40755839963 (https://github.com/cluebotng/component-configs/commits/43e1fcf7f4599d40cc9b9bd87abfeba5baede79b)
* 19:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40752366984 (https://github.com/cluebotng/component-configs/commits/b792020c142bd09c08d70ae0f40684c9c3f81079)
* 13:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40698633627 (https://github.com/cluebotng/component-configs/commits/29d35958a8230b19c67327fae4cc1a2c55f8f4f0)
* 13:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40697046893 (https://github.com/cluebotng/component-configs/commits/65c2635e0eca77c926d6ecfdd3bc4243290cb2de)
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788397 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)
* 06:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40643749649 (https://github.com/cluebotng/component-configs/commits/6aa169a5ad67427223a008319e43e00be9db16c5)
* 04:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40632642704 (https://github.com/cluebotng/component-configs/commits/611facf7667c39a7f5b1712e5b174684a0649ea4)

=== 2026-10-27 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40545550695 (https://github.com/cluebotng/component-configs/commits/9a4c6ecd52da46f16ac808838c87f2b4608e2987)
* 16:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40536663681 (https://github.com/cluebotng/component-configs/commits/d02fac71cd9a7c00d0b2cba2a245b95677c46c4d)
* 15:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40527035651 (https://github.com/cluebotng/component-configs/commits/7dd9e7d26ac0141af54d06f81163a130e5a402d3)
* 15:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40526418448 (https://github.com/cluebotng/component-configs/commits/9c1e7dfb720d9ef57fe0e066dcebe9ccd5701eaf)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508590861 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40489900248 (https://github.com/cluebotng/component-configs/commits/8007804f0273db3ecaf618e4e3723dbe1bf0fb67)
* 10:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40487986665 (https://github.com/cluebotng/component-configs/commits/7cc193ac0999f846b516b843914c70a13924f99f)
* 06:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40462539944 (https://github.com/cluebotng/component-configs/commits/e4c07d7c9925e5e434298a95070b8f40a7db75c8)
* 06:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459831650 (https://github.com/cluebotng/component-configs/commits/bfbe8dc9171d388710c1d5eface2d00809cd79dd)
* 05:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40456450428 (https://github.com/cluebotng/component-configs/commits/b0fce42989bc739f2992831884538d17c4220a07)
* 05:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453802690 (https://github.com/cluebotng/component-configs/commits/4186d7d35e3637e5f7722d66c028994026229d84)
* 05:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40453565998 (https://github.com/cluebotng/component-configs/commits/0b40c20b317be8b21d7e6b4a9c452f2be1a63558)
* 05:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518130 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40451550237 (https://github.com/cluebotng/component-configs/commits/d19590e442b664819343d4373714879e039ae49f)
* 04:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40448492483 (https://github.com/cluebotng/component-configs/commits/d09fba9de2a42159fa1c899076588e5408ccc19b)
* 04:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447571824 (https://github.com/cluebotng/component-configs/commits/7d10e42f436c05ef56f6f317648388d592c3c05f)
* 04:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375410 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446526995 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445124134 (https://github.com/cluebotng/component-configs/commits/f7026cd7409b14b6865323d50c6942fde358921e)

=== 2026-09-01 ===
* 16:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/33418019250 (https://github.com/cluebotng/component-configs/commits/c689d315eb65ad2cb2f8d0b28293864f7948ce53)

=== 2026-07-02 ===
* 21:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-20 ===
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27870870297 (https://github.com/cluebotng/component-configs/commits/331545fc215d0e72512d841be6ac751372a6c229)

=== 2026-06-16 ===
* 12:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27618465342 (https://github.com/cluebotng/component-configs/commits/9b3fcbbd479041d764ab53f0a74027e2df6df4f6)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617439342 (https://github.com/cluebotng/component-configs/commits/25d55050b7cd85801d40ba66cf804d6d4a3b51bd)
* 12:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27617061512 (https://github.com/cluebotng/component-configs/commits/4505fa21f50c25710eaf644eecdc0233f726c946)
* 12:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804749 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 12:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616668922 (https://github.com/cluebotng/component-configs/commits/1bfb0487d9d8b3b799e0d9a30269ca314046feba)
* 12:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615852687 (https://github.com/cluebotng/component-configs/commits/c9746412542e654e3ae7337a570727eb5d7195d6)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27615771202 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-15 ===
* 16:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27560286658 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27557053203 (https://github.com/cluebotng/component-configs/commits/80c3b871693d13296e1c4640105e4a8cabc9d5a4)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27556781475 (https://github.com/cluebotng/component-configs/commits/81d6af2f5fa912449070fe6ac104761023a9960b)
* 14:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554700598 (https://github.com/cluebotng/component-configs/commits/12298f8c7711b0dbc3ebe3196da055b62b307301)

=== 2026-06-10 ===
* 14:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27285126180 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27277788903 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719865829 (https://github.com/cluebotng/component-configs/commits/ea5cbd0bd54e33719b9e05b40b425784b205838f)
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719857015 (https://github.com/cluebotng/component-configs/commits/017ef6a67c2650b45476bc9d5021e7abec570d4f)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719584366 (https://github.com/cluebotng/component-configs/commits/db6b6fb5bb2d48eb49c6ceb67565398f3d58dcac)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719577987 (https://github.com/cluebotng/component-configs/commits/9d18fa5f3d4d6f4cdcec776c648bb2d93e2ba652)

=== 2026-05-29 ===
* 02:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26614336971 (https://github.com/cluebotng/component-configs/commits/622f302a735e2a285b7ac4f46e726aaf538f8d93)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26611217389 (https://github.com/cluebotng/component-configs/commits/1c99ebab25e9cbf3bd595f45368b3a5e15489941)

=== 2026-05-28 ===
* 23:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26609327945 (https://github.com/cluebotng/component-configs/commits/3772f3cb5ead5bf2f5baf68cb048123a3b98c006)
* 23:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26607745901 (https://github.com/cluebotng/component-configs/commits/4cb42c1d89330943cefe6c3f647fe33e1760d090)
* 21:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26603560881 (https://github.com/cluebotng/component-configs/commits/c78caa8aac55b55f6da890066aba0130382c840f)
* 19:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26598011849 (https://github.com/cluebotng/component-configs/commits/0c29ead3e5562572c27b58851d98362fc1222a01)
* 19:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26595835360 (https://github.com/cluebotng/component-configs/commits/b6254ea98d0e678578264c75442834eb5f9b6421)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26587853908 (https://github.com/cluebotng/component-configs/commits/e6b9e2d0e00d40589e0bebda0bbbffdc178bea3a)
* 11:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26571205214 (https://github.com/cluebotng/component-configs/commits/8f023304162d9582d5736568d9d7078c10b9041d)
* 10:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26568183905 (https://github.com/cluebotng/component-configs/commits/21b5bddbae9516620318fc278d5c3d1e3f295511)

=== 2026-05-21 ===
* 22:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256956868 (https://github.com/cluebotng/component-configs/commits/95f311a49172258bfd4d1c00d74a918a7aceab68)
* 22:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26256861316 (https://github.com/cluebotng/component-configs/commits/e4c68154f62c8846320103abb0e97af63b20f9b1)
* 21:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254813158 (https://github.com/cluebotng/component-configs/commits/ebf46d18f22eee769feeeffd6f4603ec4f320bbb)
* 21:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26254787328 (https://github.com/cluebotng/component-configs/commits/80d9748516b3523cecfc76bd85d662fe91dde0c4)

=== 2026-05-19 ===
* 02:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26072649347 (https://github.com/cluebotng/component-configs/commits/94b7da488f5c23df52fa2e750438a3a80f589f3c)
* 02:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071765151 (https://github.com/cluebotng/component-configs/commits/5184d69b34b8d2442df1fbbd652535f5890b4804)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26070598819 (https://github.com/cluebotng/component-configs/commits/c5378a5646858d02f07ababdb2db2955000ab187)

=== 2026-05-18 ===
* 23:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26066535621 (https://github.com/cluebotng/component-configs/commits/2b0b0bbd4e0a727a468343823d4343fe7a082ef9)
* 23:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26065893095 (https://github.com/cluebotng/component-configs/commits/0bf6e1847388fd90f2c03acdc1b80d388cb437e7)

=== 2026-05-17 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986256508 (https://github.com/cluebotng/component-configs/commits/fcc6e35d16a8ccd601ed7f37bbafcc9b26b8f80f)
* 01:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977577391 (https://github.com/cluebotng/component-configs/commits/b7ea6acca0022ec902075282f7fd686162948bc8)
* 01:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>5iitvv81c65cluy1vufasvj3wh5tu0m</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-trainer/SAL</title>
    <ns>498</ns>
    <id>459244</id>
    <revision>
      <id>2478976</id>
      <parentid>2478957</parentid>
      <timestamp>2026-11-07T12:31:57Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819815 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478976</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="36739" sha1="rlq5pcou2huxzkm9v6eppn3j98204a2" xml:space="preserve">=== 2026-11-07 ===
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819815 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251090 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 18:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42272751673 (https://github.com/cluebotng/component-configs/commits/c17542a324406771d842160c3b55a45fac0d1dae)
* 16:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695552 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249188398 (https://github.com/cluebotng/component-configs/commits/276675d9fde45fb79a69d85827a61fea149951b7)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248468132 (https://github.com/cluebotng/component-configs/commits/973b64f18651ed5b18cf908235604e70028861c8)
* 15:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42240867017 (https://github.com/cluebotng/component-configs/commits/482c4765fb8b7f5b70651b620c2191c412c78234)
* 15:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42240676663 (https://github.com/cluebotng/component-configs/commits/4f4442799837e798ff4c86d87a85f93dd57a70e6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234420334 (https://github.com/cluebotng/component-configs/commits/0c4dbb64c592db6c7c12afbcc69dec8082f6c60b)
* 14:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833462 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233360609 (https://github.com/cluebotng/component-configs/commits/22a72d8f43f6861bbb9ac592cb9af892dd95f765)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233249218 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188055 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42215994580 (https://github.com/cluebotng/component-configs/commits/e3ea44488c33b1bd4dd63a5c9d9c3ef4a58c3406)
* 07:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177719434 (https://github.com/cluebotng/component-configs/commits/66d0b6ee6547ac11df039d6d7e288c9f0acac36b)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270563 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297634 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 03:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618016 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099049001 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974709611 (https://github.com/cluebotng/component-configs/commits/5efbb82f84986437a20d11414253c6eb8926c8d2)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300499 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 03:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41961409257 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41865423884 (https://github.com/cluebotng/component-configs/commits/86b7159004732c324a71b91e87663d2e1cea32df)
* 03:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208826 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764161174 (https://github.com/cluebotng/component-configs/commits/efdda3a013777f4e684be77dc6e05704036d4f0a)
* 00:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119449 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693130037 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894047 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194741 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616245 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586263970 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567113718 (https://github.com/cluebotng/component-configs/commits/10880256092163e5d6d4d97b1d2a0453e51c1f2c)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434982 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529420 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190339868 (https://github.com/cluebotng/component-configs/commits/c258f6030d4d454fbda316e043577097ae61d646)
* 09:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726816 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 08:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002297 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595348 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610807 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626137 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41137073295 (https://github.com/cluebotng/component-configs/commits/dbe1aec775bd0f29e13e52e8e5820ade20307178)
* 21:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558384 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125890120 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022564 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696917 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902351489 (https://github.com/cluebotng/component-configs/commits/a712571c820f2a1ea767bd46ea74112426fb56bc)
* 15:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901778442 (https://github.com/cluebotng/component-configs/commits/abdf069b6a9204f3fe2796741e1beb93d89a5d07)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900617668 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898272930 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266178 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 09:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40665456675 (https://github.com/cluebotng/component-configs/commits/cbd8bb224c15c6dacdf93632ef29a11586a0b50b)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788352 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846848 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 14:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517885090 (https://github.com/cluebotng/component-configs/commits/fbcf68ea18c187fbb8af37ed04a674837d121903)
* 14:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517784751 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591039 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 09:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40484261158 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540079 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457189661 (https://github.com/cluebotng/component-configs/commits/30587ffc7ef251c2e1a6bcd1d55b7a3b2493d65c)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798034 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518133 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375409 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527142 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221759 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)
* 03:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444706514 (https://github.com/cluebotng/component-configs/commits/a9515adb6b57ca73316c1c6b8559bc6b49255d7f)
* 03:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183977 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412786 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506484 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 17:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849962 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634738411 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634024663 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615771226 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125894 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788620 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276698067 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510520 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411775 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339656 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225357080 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156826 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-03 ===
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858620 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884839872 (https://github.com/cluebotng/component-configs/commits/cc341cdd381588d285ccbc586ab019ece652d3ce)

=== 2026-05-31 ===
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864789 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858628 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719586315 (https://github.com/cluebotng/component-configs/commits/30600f798367d8fe4124ab5042ddc70122366500)

=== 2026-05-29 ===
* 00:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26610547573 (https://github.com/cluebotng/component-configs/commits/d647b7c087dff8de798d78a98082469fa41e7712)

=== 2026-05-28 ===
* 11:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26572448836 (https://github.com/cluebotng/component-configs/commits/fdea8d6284d14cd9ed835ab73506d0750af62de2)

=== 2026-05-21 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26241143565 (https://github.com/cluebotng/component-configs/commits/76187b1d77a1454297932d833d668fafc6490c10)

=== 2026-05-19 ===
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071084518 (https://github.com/cluebotng/component-configs/commits/6831fe17857c84db1afb1023198b7b11d444f69f)

=== 2026-05-17 ===
* 08:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25985888959 (https://github.com/cluebotng/component-configs/commits/591f7870cefcc3c295b1136fa6afb385738dd863)

=== 2026-05-14 ===
* 17:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda)

=== 2026-05-13 ===
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771881181 (https://github.com/cluebotng/component-configs/commits/d180b98d98a8b5c89115a2080375145455ab4730)

=== 2026-05-12 ===
* 10:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25729275449 (https://github.com/cluebotng/component-configs/commits/ee80e1a674aa4d62ec0736286534130973827ac8)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848224 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)
* 09:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842081 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480610 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)

=== 2026-05-06 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25459711948 (https://github.com/cluebotng/component-configs/commits/3e0d938cc4bbeb5069a3e0b0d864df7eabf78c23)
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451879961 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)
* 16:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25447977391 (https://github.com/cluebotng/component-configs/commits/bd39eba234686d41e4d889def1ed91fc28e8fdf6)

=== 2026-04-24 ===
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24913659688 (https://github.com/cluebotng/component-configs/commits/c5ad64125c64f5db76d9a8141163d380f1c2d4f0)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591292 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)
* 00:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540900579 (https://github.com/cluebotng/component-configs/commits/bfd1f6b6b742e082dbbc587464ca135bfe827765)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007472 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)
* 09:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357383991 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442397 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897956 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249371602 (https://github.com/cluebotng/component-configs/commits/acce8e3abd2da42aef50053d1c1126daec711da1)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248696057 (https://github.com/cluebotng/component-configs/commits/310efa6d212253bd549cddd16d85f4167b26f684)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247621716 (https://github.com/cluebotng/component-configs/commits/2e25667a8eb92011273ca47b217f3546da48cae9)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582931 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093268 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659602695 (https://github.com/cluebotng/component-configs/commits/6f1ef446ee988193ccd8d9c62eb4841267976159)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537319 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959549 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556336 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)
* 16:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383552275 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-14 ===
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358864102 (https://github.com/cluebotng/component-configs/commits/b9059392c8da641c97c97919e1d017bf0251914f)
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733947 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 22:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19347680045 (https://github.com/cluebotng/component-configs/commits/5de2267efa622608c868e4cb66658a4f367aa3b0)
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862269 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)
* 19:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342998586 (https://github.com/cluebotng/component-configs/commits/17a2abf82e9a2076113891e4ae3d5159c048b940)
* 19:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342782944 (https://github.com/cluebotng/component-configs/commits/fed3ef035303fd54334c5a7932d23a43e381492e)
* 18:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342079797 (https://github.com/cluebotng/component-configs/commits/b8c173ced436297aaa9a09b2ef9a4b814d696a30)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341777629 (https://github.com/cluebotng/component-configs/commits/afb04303301910afc2a7692e2e5676540a27c38c)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341671161 (https://github.com/cluebotng/component-configs/commits/84f054bb9402daea356833fbd2d606f4d2b44b48)
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341445508 (https://github.com/cluebotng/component-configs/commits/2a497ee04f7182e6d114d1aafa79a99e29ff6d22)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341306596 (https://github.com/cluebotng/component-configs/commits/6e3dd964163e25f7115b74c7b132316745613729)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341017921 (https://github.com/cluebotng/component-configs/commits/b28d25e5b98fe60c2b85138e7d1a2a52cee03383)
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340976225 (https://github.com/cluebotng/component-configs/commits/8c3a4f743978487c10274534162bb1a7a2ceb8a8)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340717614 (https://github.com/cluebotng/component-configs/commits/a2ac5e2b12d80df59f7f37209ed8943fe471992a)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340563565 (https://github.com/cluebotng/component-configs/commits/a5ea7ec11c7d2a7ced4f29df1f2a742f52dbf777)
* 15:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19337204923 (https://github.com/cluebotng/component-configs/commits/d5247b0e2c72c4426888761d70d73f59ba7362a5)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642882 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270313845 (https://github.com/cluebotng/component-configs/commits/900a30b8983bc80a537330fa4345e7952f2081b4)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264807 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19267535662 (https://github.com/cluebotng/component-configs/commits/eae37aa10505a2ac3c773cdd4f85b7222cdae836)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266909480 (https://github.com/cluebotng/component-configs/commits/bc0dd19078113c4e1cbe90a059fe26932fb70a29)

=== 2025-10-07 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304356335 (https://github.com/cluebotng/component-configs/commits/49f902fb1380f69c5e0156466832b031a9c228ee)

=== 2025-10-06 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272376354 (https://github.com/cluebotng/component-configs/commits/449c7d3c68ade43ce238b9b335182dc50f929511)

=== 2025-10-05 ===
* 06:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255175939 (https://github.com/cluebotng/component-configs/commits/f7e65671f2deb536f52fdf0c1db7ec80709a76da)

=== 2025-10-01 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18153820283 (https://github.com/cluebotng/component-configs/commits/9f9e9abc422c8a74207416afbaf81b72509e91ce)

=== 2025-09-30 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18121221077 (https://github.com/cluebotng/component-configs/commits/11eb3da0c88160bca0ec60e55dadad7e928e40f3)

=== 2025-09-29 ===
* 16:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101421 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101913411 (https://github.com/cluebotng/component-configs/commits/1b6389ae74b8974d6f49591c0abf14a8da974c4b)
* 15:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460176 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217564 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)

=== 2025-09-26 ===
* 06:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030092297 (https://github.com/cluebotng/component-configs/commits/2516589b458034b7c2540c998b29debeb63e82df)

=== 2025-09-25 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801857 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221974 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782304 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677303 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531079 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008148799 (https://github.com/cluebotng/component-configs/commits/0bb2ca770839346853b6a8440927a8afde70c94d)

=== 2025-09-24 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986145339 (https://github.com/cluebotng/component-configs/commits/820291b11edc6256cfa07e9a4a73677df86e52d8)
* 18:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986015781 (https://github.com/cluebotng/component-configs/commits/b0f088ec49a2156a18ad7d78d18640f6e8fe943c)
* 18:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985989347 (https://github.com/cluebotng/component-configs/commits/38b2f2235d143f0522d560a93c04e50e8fa38c77)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985728221 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985446508 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985386789 (https://github.com/cluebotng/component-configs/commits/a1633fc7ecf1bd4a310fcb03d02d0da81212427a)
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333827 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198515 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077117 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820831 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983737370 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 20:30 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17927644810
* 19:02 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925536572


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>rlq5pcou2huxzkm9v6eppn3j98204a2</sha1>
    </revision>
    <revision>
      <id>2478985</id>
      <parentid>2478976</parentid>
      <timestamp>2026-11-07T12:36:45Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856569 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)</comment>
      <origin>2478985</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="36945" sha1="nhowamjzfm0yozd9cn4c76jvt4d69nm" xml:space="preserve">=== 2026-11-07 ===
* 12:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856569 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819815 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251090 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 18:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42272751673 (https://github.com/cluebotng/component-configs/commits/c17542a324406771d842160c3b55a45fac0d1dae)
* 16:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695552 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249188398 (https://github.com/cluebotng/component-configs/commits/276675d9fde45fb79a69d85827a61fea149951b7)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248468132 (https://github.com/cluebotng/component-configs/commits/973b64f18651ed5b18cf908235604e70028861c8)
* 15:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42240867017 (https://github.com/cluebotng/component-configs/commits/482c4765fb8b7f5b70651b620c2191c412c78234)
* 15:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42240676663 (https://github.com/cluebotng/component-configs/commits/4f4442799837e798ff4c86d87a85f93dd57a70e6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234420334 (https://github.com/cluebotng/component-configs/commits/0c4dbb64c592db6c7c12afbcc69dec8082f6c60b)
* 14:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833462 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233360609 (https://github.com/cluebotng/component-configs/commits/22a72d8f43f6861bbb9ac592cb9af892dd95f765)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233249218 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188055 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42215994580 (https://github.com/cluebotng/component-configs/commits/e3ea44488c33b1bd4dd63a5c9d9c3ef4a58c3406)
* 07:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177719434 (https://github.com/cluebotng/component-configs/commits/66d0b6ee6547ac11df039d6d7e288c9f0acac36b)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270563 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297634 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 03:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618016 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099049001 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974709611 (https://github.com/cluebotng/component-configs/commits/5efbb82f84986437a20d11414253c6eb8926c8d2)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300499 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 03:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41961409257 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41865423884 (https://github.com/cluebotng/component-configs/commits/86b7159004732c324a71b91e87663d2e1cea32df)
* 03:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208826 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764161174 (https://github.com/cluebotng/component-configs/commits/efdda3a013777f4e684be77dc6e05704036d4f0a)
* 00:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119449 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693130037 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894047 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194741 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616245 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586263970 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567113718 (https://github.com/cluebotng/component-configs/commits/10880256092163e5d6d4d97b1d2a0453e51c1f2c)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434982 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529420 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190339868 (https://github.com/cluebotng/component-configs/commits/c258f6030d4d454fbda316e043577097ae61d646)
* 09:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726816 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 08:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002297 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595348 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610807 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626137 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41137073295 (https://github.com/cluebotng/component-configs/commits/dbe1aec775bd0f29e13e52e8e5820ade20307178)
* 21:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558384 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125890120 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022564 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696917 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902351489 (https://github.com/cluebotng/component-configs/commits/a712571c820f2a1ea767bd46ea74112426fb56bc)
* 15:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901778442 (https://github.com/cluebotng/component-configs/commits/abdf069b6a9204f3fe2796741e1beb93d89a5d07)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900617668 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898272930 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266178 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 09:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40665456675 (https://github.com/cluebotng/component-configs/commits/cbd8bb224c15c6dacdf93632ef29a11586a0b50b)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788352 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846848 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 14:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517885090 (https://github.com/cluebotng/component-configs/commits/fbcf68ea18c187fbb8af37ed04a674837d121903)
* 14:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517784751 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591039 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 09:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40484261158 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540079 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457189661 (https://github.com/cluebotng/component-configs/commits/30587ffc7ef251c2e1a6bcd1d55b7a3b2493d65c)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798034 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518133 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375409 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527142 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221759 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)
* 03:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444706514 (https://github.com/cluebotng/component-configs/commits/a9515adb6b57ca73316c1c6b8559bc6b49255d7f)
* 03:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183977 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412786 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506484 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 17:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849962 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634738411 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634024663 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615771226 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125894 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788620 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276698067 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510520 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411775 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339656 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225357080 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156826 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-03 ===
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858620 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884839872 (https://github.com/cluebotng/component-configs/commits/cc341cdd381588d285ccbc586ab019ece652d3ce)

=== 2026-05-31 ===
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864789 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858628 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719586315 (https://github.com/cluebotng/component-configs/commits/30600f798367d8fe4124ab5042ddc70122366500)

=== 2026-05-29 ===
* 00:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26610547573 (https://github.com/cluebotng/component-configs/commits/d647b7c087dff8de798d78a98082469fa41e7712)

=== 2026-05-28 ===
* 11:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26572448836 (https://github.com/cluebotng/component-configs/commits/fdea8d6284d14cd9ed835ab73506d0750af62de2)

=== 2026-05-21 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26241143565 (https://github.com/cluebotng/component-configs/commits/76187b1d77a1454297932d833d668fafc6490c10)

=== 2026-05-19 ===
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071084518 (https://github.com/cluebotng/component-configs/commits/6831fe17857c84db1afb1023198b7b11d444f69f)

=== 2026-05-17 ===
* 08:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25985888959 (https://github.com/cluebotng/component-configs/commits/591f7870cefcc3c295b1136fa6afb385738dd863)

=== 2026-05-14 ===
* 17:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda)

=== 2026-05-13 ===
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771881181 (https://github.com/cluebotng/component-configs/commits/d180b98d98a8b5c89115a2080375145455ab4730)

=== 2026-05-12 ===
* 10:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25729275449 (https://github.com/cluebotng/component-configs/commits/ee80e1a674aa4d62ec0736286534130973827ac8)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848224 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)
* 09:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842081 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480610 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)

=== 2026-05-06 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25459711948 (https://github.com/cluebotng/component-configs/commits/3e0d938cc4bbeb5069a3e0b0d864df7eabf78c23)
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451879961 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)
* 16:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25447977391 (https://github.com/cluebotng/component-configs/commits/bd39eba234686d41e4d889def1ed91fc28e8fdf6)

=== 2026-04-24 ===
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24913659688 (https://github.com/cluebotng/component-configs/commits/c5ad64125c64f5db76d9a8141163d380f1c2d4f0)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591292 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)
* 00:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540900579 (https://github.com/cluebotng/component-configs/commits/bfd1f6b6b742e082dbbc587464ca135bfe827765)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007472 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)
* 09:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357383991 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442397 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897956 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249371602 (https://github.com/cluebotng/component-configs/commits/acce8e3abd2da42aef50053d1c1126daec711da1)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248696057 (https://github.com/cluebotng/component-configs/commits/310efa6d212253bd549cddd16d85f4167b26f684)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247621716 (https://github.com/cluebotng/component-configs/commits/2e25667a8eb92011273ca47b217f3546da48cae9)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582931 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093268 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659602695 (https://github.com/cluebotng/component-configs/commits/6f1ef446ee988193ccd8d9c62eb4841267976159)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537319 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959549 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556336 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)
* 16:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383552275 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-14 ===
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358864102 (https://github.com/cluebotng/component-configs/commits/b9059392c8da641c97c97919e1d017bf0251914f)
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733947 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 22:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19347680045 (https://github.com/cluebotng/component-configs/commits/5de2267efa622608c868e4cb66658a4f367aa3b0)
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862269 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)
* 19:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342998586 (https://github.com/cluebotng/component-configs/commits/17a2abf82e9a2076113891e4ae3d5159c048b940)
* 19:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342782944 (https://github.com/cluebotng/component-configs/commits/fed3ef035303fd54334c5a7932d23a43e381492e)
* 18:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342079797 (https://github.com/cluebotng/component-configs/commits/b8c173ced436297aaa9a09b2ef9a4b814d696a30)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341777629 (https://github.com/cluebotng/component-configs/commits/afb04303301910afc2a7692e2e5676540a27c38c)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341671161 (https://github.com/cluebotng/component-configs/commits/84f054bb9402daea356833fbd2d606f4d2b44b48)
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341445508 (https://github.com/cluebotng/component-configs/commits/2a497ee04f7182e6d114d1aafa79a99e29ff6d22)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341306596 (https://github.com/cluebotng/component-configs/commits/6e3dd964163e25f7115b74c7b132316745613729)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341017921 (https://github.com/cluebotng/component-configs/commits/b28d25e5b98fe60c2b85138e7d1a2a52cee03383)
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340976225 (https://github.com/cluebotng/component-configs/commits/8c3a4f743978487c10274534162bb1a7a2ceb8a8)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340717614 (https://github.com/cluebotng/component-configs/commits/a2ac5e2b12d80df59f7f37209ed8943fe471992a)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340563565 (https://github.com/cluebotng/component-configs/commits/a5ea7ec11c7d2a7ced4f29df1f2a742f52dbf777)
* 15:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19337204923 (https://github.com/cluebotng/component-configs/commits/d5247b0e2c72c4426888761d70d73f59ba7362a5)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642882 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270313845 (https://github.com/cluebotng/component-configs/commits/900a30b8983bc80a537330fa4345e7952f2081b4)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264807 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19267535662 (https://github.com/cluebotng/component-configs/commits/eae37aa10505a2ac3c773cdd4f85b7222cdae836)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266909480 (https://github.com/cluebotng/component-configs/commits/bc0dd19078113c4e1cbe90a059fe26932fb70a29)

=== 2025-10-07 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304356335 (https://github.com/cluebotng/component-configs/commits/49f902fb1380f69c5e0156466832b031a9c228ee)

=== 2025-10-06 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272376354 (https://github.com/cluebotng/component-configs/commits/449c7d3c68ade43ce238b9b335182dc50f929511)

=== 2025-10-05 ===
* 06:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255175939 (https://github.com/cluebotng/component-configs/commits/f7e65671f2deb536f52fdf0c1db7ec80709a76da)

=== 2025-10-01 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18153820283 (https://github.com/cluebotng/component-configs/commits/9f9e9abc422c8a74207416afbaf81b72509e91ce)

=== 2025-09-30 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18121221077 (https://github.com/cluebotng/component-configs/commits/11eb3da0c88160bca0ec60e55dadad7e928e40f3)

=== 2025-09-29 ===
* 16:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101421 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101913411 (https://github.com/cluebotng/component-configs/commits/1b6389ae74b8974d6f49591c0abf14a8da974c4b)
* 15:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460176 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217564 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)

=== 2025-09-26 ===
* 06:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030092297 (https://github.com/cluebotng/component-configs/commits/2516589b458034b7c2540c998b29debeb63e82df)

=== 2025-09-25 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801857 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221974 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782304 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677303 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531079 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008148799 (https://github.com/cluebotng/component-configs/commits/0bb2ca770839346853b6a8440927a8afde70c94d)

=== 2025-09-24 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986145339 (https://github.com/cluebotng/component-configs/commits/820291b11edc6256cfa07e9a4a73677df86e52d8)
* 18:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986015781 (https://github.com/cluebotng/component-configs/commits/b0f088ec49a2156a18ad7d78d18640f6e8fe943c)
* 18:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985989347 (https://github.com/cluebotng/component-configs/commits/38b2f2235d143f0522d560a93c04e50e8fa38c77)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985728221 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985446508 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985386789 (https://github.com/cluebotng/component-configs/commits/a1633fc7ecf1bd4a310fcb03d02d0da81212427a)
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333827 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198515 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077117 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820831 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983737370 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 20:30 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17927644810
* 19:02 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925536572


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>nhowamjzfm0yozd9cn4c76jvt4d69nm</sha1>
    </revision>
    <revision>
      <id>2478995</id>
      <parentid>2478985</parentid>
      <timestamp>2026-11-07T12:41:13Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017679 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)</comment>
      <origin>2478995</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="37151" sha1="6kkob56a4ksvngsz1h7cm7ss5b15nk3" xml:space="preserve">=== 2026-11-07 ===
* 12:41 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017679 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856569 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819815 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251090 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 18:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42272751673 (https://github.com/cluebotng/component-configs/commits/c17542a324406771d842160c3b55a45fac0d1dae)
* 16:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695552 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249188398 (https://github.com/cluebotng/component-configs/commits/276675d9fde45fb79a69d85827a61fea149951b7)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248468132 (https://github.com/cluebotng/component-configs/commits/973b64f18651ed5b18cf908235604e70028861c8)
* 15:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42240867017 (https://github.com/cluebotng/component-configs/commits/482c4765fb8b7f5b70651b620c2191c412c78234)
* 15:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42240676663 (https://github.com/cluebotng/component-configs/commits/4f4442799837e798ff4c86d87a85f93dd57a70e6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234420334 (https://github.com/cluebotng/component-configs/commits/0c4dbb64c592db6c7c12afbcc69dec8082f6c60b)
* 14:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833462 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233360609 (https://github.com/cluebotng/component-configs/commits/22a72d8f43f6861bbb9ac592cb9af892dd95f765)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233249218 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188055 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42215994580 (https://github.com/cluebotng/component-configs/commits/e3ea44488c33b1bd4dd63a5c9d9c3ef4a58c3406)
* 07:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177719434 (https://github.com/cluebotng/component-configs/commits/66d0b6ee6547ac11df039d6d7e288c9f0acac36b)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270563 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297634 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 03:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618016 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099049001 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974709611 (https://github.com/cluebotng/component-configs/commits/5efbb82f84986437a20d11414253c6eb8926c8d2)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300499 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 03:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41961409257 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41865423884 (https://github.com/cluebotng/component-configs/commits/86b7159004732c324a71b91e87663d2e1cea32df)
* 03:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208826 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764161174 (https://github.com/cluebotng/component-configs/commits/efdda3a013777f4e684be77dc6e05704036d4f0a)
* 00:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119449 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693130037 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894047 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194741 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616245 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586263970 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567113718 (https://github.com/cluebotng/component-configs/commits/10880256092163e5d6d4d97b1d2a0453e51c1f2c)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434982 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529420 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190339868 (https://github.com/cluebotng/component-configs/commits/c258f6030d4d454fbda316e043577097ae61d646)
* 09:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726816 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 08:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002297 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595348 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610807 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626137 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41137073295 (https://github.com/cluebotng/component-configs/commits/dbe1aec775bd0f29e13e52e8e5820ade20307178)
* 21:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558384 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125890120 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022564 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696917 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902351489 (https://github.com/cluebotng/component-configs/commits/a712571c820f2a1ea767bd46ea74112426fb56bc)
* 15:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901778442 (https://github.com/cluebotng/component-configs/commits/abdf069b6a9204f3fe2796741e1beb93d89a5d07)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900617668 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898272930 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266178 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 09:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40665456675 (https://github.com/cluebotng/component-configs/commits/cbd8bb224c15c6dacdf93632ef29a11586a0b50b)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788352 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846848 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 14:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517885090 (https://github.com/cluebotng/component-configs/commits/fbcf68ea18c187fbb8af37ed04a674837d121903)
* 14:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517784751 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591039 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 09:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40484261158 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540079 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457189661 (https://github.com/cluebotng/component-configs/commits/30587ffc7ef251c2e1a6bcd1d55b7a3b2493d65c)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798034 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518133 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375409 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527142 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221759 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)
* 03:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444706514 (https://github.com/cluebotng/component-configs/commits/a9515adb6b57ca73316c1c6b8559bc6b49255d7f)
* 03:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183977 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412786 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506484 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 17:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849962 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634738411 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634024663 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615771226 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125894 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788620 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276698067 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510520 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411775 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339656 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225357080 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156826 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-03 ===
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858620 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884839872 (https://github.com/cluebotng/component-configs/commits/cc341cdd381588d285ccbc586ab019ece652d3ce)

=== 2026-05-31 ===
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864789 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858628 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719586315 (https://github.com/cluebotng/component-configs/commits/30600f798367d8fe4124ab5042ddc70122366500)

=== 2026-05-29 ===
* 00:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26610547573 (https://github.com/cluebotng/component-configs/commits/d647b7c087dff8de798d78a98082469fa41e7712)

=== 2026-05-28 ===
* 11:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26572448836 (https://github.com/cluebotng/component-configs/commits/fdea8d6284d14cd9ed835ab73506d0750af62de2)

=== 2026-05-21 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26241143565 (https://github.com/cluebotng/component-configs/commits/76187b1d77a1454297932d833d668fafc6490c10)

=== 2026-05-19 ===
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071084518 (https://github.com/cluebotng/component-configs/commits/6831fe17857c84db1afb1023198b7b11d444f69f)

=== 2026-05-17 ===
* 08:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25985888959 (https://github.com/cluebotng/component-configs/commits/591f7870cefcc3c295b1136fa6afb385738dd863)

=== 2026-05-14 ===
* 17:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda)

=== 2026-05-13 ===
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771881181 (https://github.com/cluebotng/component-configs/commits/d180b98d98a8b5c89115a2080375145455ab4730)

=== 2026-05-12 ===
* 10:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25729275449 (https://github.com/cluebotng/component-configs/commits/ee80e1a674aa4d62ec0736286534130973827ac8)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848224 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)
* 09:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842081 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480610 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)

=== 2026-05-06 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25459711948 (https://github.com/cluebotng/component-configs/commits/3e0d938cc4bbeb5069a3e0b0d864df7eabf78c23)
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451879961 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)
* 16:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25447977391 (https://github.com/cluebotng/component-configs/commits/bd39eba234686d41e4d889def1ed91fc28e8fdf6)

=== 2026-04-24 ===
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24913659688 (https://github.com/cluebotng/component-configs/commits/c5ad64125c64f5db76d9a8141163d380f1c2d4f0)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591292 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)
* 00:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540900579 (https://github.com/cluebotng/component-configs/commits/bfd1f6b6b742e082dbbc587464ca135bfe827765)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007472 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)
* 09:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357383991 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442397 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897956 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249371602 (https://github.com/cluebotng/component-configs/commits/acce8e3abd2da42aef50053d1c1126daec711da1)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248696057 (https://github.com/cluebotng/component-configs/commits/310efa6d212253bd549cddd16d85f4167b26f684)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247621716 (https://github.com/cluebotng/component-configs/commits/2e25667a8eb92011273ca47b217f3546da48cae9)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582931 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093268 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659602695 (https://github.com/cluebotng/component-configs/commits/6f1ef446ee988193ccd8d9c62eb4841267976159)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537319 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959549 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556336 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)
* 16:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383552275 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-14 ===
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358864102 (https://github.com/cluebotng/component-configs/commits/b9059392c8da641c97c97919e1d017bf0251914f)
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733947 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 22:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19347680045 (https://github.com/cluebotng/component-configs/commits/5de2267efa622608c868e4cb66658a4f367aa3b0)
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862269 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)
* 19:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342998586 (https://github.com/cluebotng/component-configs/commits/17a2abf82e9a2076113891e4ae3d5159c048b940)
* 19:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342782944 (https://github.com/cluebotng/component-configs/commits/fed3ef035303fd54334c5a7932d23a43e381492e)
* 18:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342079797 (https://github.com/cluebotng/component-configs/commits/b8c173ced436297aaa9a09b2ef9a4b814d696a30)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341777629 (https://github.com/cluebotng/component-configs/commits/afb04303301910afc2a7692e2e5676540a27c38c)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341671161 (https://github.com/cluebotng/component-configs/commits/84f054bb9402daea356833fbd2d606f4d2b44b48)
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341445508 (https://github.com/cluebotng/component-configs/commits/2a497ee04f7182e6d114d1aafa79a99e29ff6d22)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341306596 (https://github.com/cluebotng/component-configs/commits/6e3dd964163e25f7115b74c7b132316745613729)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341017921 (https://github.com/cluebotng/component-configs/commits/b28d25e5b98fe60c2b85138e7d1a2a52cee03383)
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340976225 (https://github.com/cluebotng/component-configs/commits/8c3a4f743978487c10274534162bb1a7a2ceb8a8)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340717614 (https://github.com/cluebotng/component-configs/commits/a2ac5e2b12d80df59f7f37209ed8943fe471992a)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340563565 (https://github.com/cluebotng/component-configs/commits/a5ea7ec11c7d2a7ced4f29df1f2a742f52dbf777)
* 15:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19337204923 (https://github.com/cluebotng/component-configs/commits/d5247b0e2c72c4426888761d70d73f59ba7362a5)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642882 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270313845 (https://github.com/cluebotng/component-configs/commits/900a30b8983bc80a537330fa4345e7952f2081b4)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264807 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19267535662 (https://github.com/cluebotng/component-configs/commits/eae37aa10505a2ac3c773cdd4f85b7222cdae836)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266909480 (https://github.com/cluebotng/component-configs/commits/bc0dd19078113c4e1cbe90a059fe26932fb70a29)

=== 2025-10-07 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304356335 (https://github.com/cluebotng/component-configs/commits/49f902fb1380f69c5e0156466832b031a9c228ee)

=== 2025-10-06 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272376354 (https://github.com/cluebotng/component-configs/commits/449c7d3c68ade43ce238b9b335182dc50f929511)

=== 2025-10-05 ===
* 06:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255175939 (https://github.com/cluebotng/component-configs/commits/f7e65671f2deb536f52fdf0c1db7ec80709a76da)

=== 2025-10-01 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18153820283 (https://github.com/cluebotng/component-configs/commits/9f9e9abc422c8a74207416afbaf81b72509e91ce)

=== 2025-09-30 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18121221077 (https://github.com/cluebotng/component-configs/commits/11eb3da0c88160bca0ec60e55dadad7e928e40f3)

=== 2025-09-29 ===
* 16:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101421 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101913411 (https://github.com/cluebotng/component-configs/commits/1b6389ae74b8974d6f49591c0abf14a8da974c4b)
* 15:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460176 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217564 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)

=== 2025-09-26 ===
* 06:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030092297 (https://github.com/cluebotng/component-configs/commits/2516589b458034b7c2540c998b29debeb63e82df)

=== 2025-09-25 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801857 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221974 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782304 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677303 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531079 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008148799 (https://github.com/cluebotng/component-configs/commits/0bb2ca770839346853b6a8440927a8afde70c94d)

=== 2025-09-24 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986145339 (https://github.com/cluebotng/component-configs/commits/820291b11edc6256cfa07e9a4a73677df86e52d8)
* 18:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986015781 (https://github.com/cluebotng/component-configs/commits/b0f088ec49a2156a18ad7d78d18640f6e8fe943c)
* 18:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985989347 (https://github.com/cluebotng/component-configs/commits/38b2f2235d143f0522d560a93c04e50e8fa38c77)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985728221 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985446508 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985386789 (https://github.com/cluebotng/component-configs/commits/a1633fc7ecf1bd4a310fcb03d02d0da81212427a)
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333827 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198515 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077117 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820831 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983737370 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 20:30 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17927644810
* 19:02 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925536572


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>6kkob56a4ksvngsz1h7cm7ss5b15nk3</sha1>
    </revision>
    <revision>
      <id>2479000</id>
      <parentid>2478995</parentid>
      <timestamp>2026-11-07T12:44:49Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6)</comment>
      <origin>2479000</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="37357" sha1="qr996efqgo9eb0ozxcfzah7zcniqiwp" xml:space="preserve">=== 2026-11-07 ===
* 12:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6)
* 12:41 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017679 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856569 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819815 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251090 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 18:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42272751673 (https://github.com/cluebotng/component-configs/commits/c17542a324406771d842160c3b55a45fac0d1dae)
* 16:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695552 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249188398 (https://github.com/cluebotng/component-configs/commits/276675d9fde45fb79a69d85827a61fea149951b7)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248468132 (https://github.com/cluebotng/component-configs/commits/973b64f18651ed5b18cf908235604e70028861c8)
* 15:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42240867017 (https://github.com/cluebotng/component-configs/commits/482c4765fb8b7f5b70651b620c2191c412c78234)
* 15:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42240676663 (https://github.com/cluebotng/component-configs/commits/4f4442799837e798ff4c86d87a85f93dd57a70e6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234420334 (https://github.com/cluebotng/component-configs/commits/0c4dbb64c592db6c7c12afbcc69dec8082f6c60b)
* 14:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833462 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233360609 (https://github.com/cluebotng/component-configs/commits/22a72d8f43f6861bbb9ac592cb9af892dd95f765)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233249218 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188055 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42215994580 (https://github.com/cluebotng/component-configs/commits/e3ea44488c33b1bd4dd63a5c9d9c3ef4a58c3406)
* 07:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177719434 (https://github.com/cluebotng/component-configs/commits/66d0b6ee6547ac11df039d6d7e288c9f0acac36b)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270563 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297634 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 03:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618016 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099049001 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974709611 (https://github.com/cluebotng/component-configs/commits/5efbb82f84986437a20d11414253c6eb8926c8d2)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300499 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 03:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41961409257 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41865423884 (https://github.com/cluebotng/component-configs/commits/86b7159004732c324a71b91e87663d2e1cea32df)
* 03:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208826 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764161174 (https://github.com/cluebotng/component-configs/commits/efdda3a013777f4e684be77dc6e05704036d4f0a)
* 00:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119449 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693130037 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894047 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194741 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616245 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586263970 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567113718 (https://github.com/cluebotng/component-configs/commits/10880256092163e5d6d4d97b1d2a0453e51c1f2c)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434982 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529420 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190339868 (https://github.com/cluebotng/component-configs/commits/c258f6030d4d454fbda316e043577097ae61d646)
* 09:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726816 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 08:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002297 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595348 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610807 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626137 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41137073295 (https://github.com/cluebotng/component-configs/commits/dbe1aec775bd0f29e13e52e8e5820ade20307178)
* 21:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558384 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125890120 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022564 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696917 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902351489 (https://github.com/cluebotng/component-configs/commits/a712571c820f2a1ea767bd46ea74112426fb56bc)
* 15:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901778442 (https://github.com/cluebotng/component-configs/commits/abdf069b6a9204f3fe2796741e1beb93d89a5d07)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900617668 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898272930 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266178 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 09:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40665456675 (https://github.com/cluebotng/component-configs/commits/cbd8bb224c15c6dacdf93632ef29a11586a0b50b)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788352 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846848 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 14:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517885090 (https://github.com/cluebotng/component-configs/commits/fbcf68ea18c187fbb8af37ed04a674837d121903)
* 14:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517784751 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591039 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 09:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40484261158 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540079 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457189661 (https://github.com/cluebotng/component-configs/commits/30587ffc7ef251c2e1a6bcd1d55b7a3b2493d65c)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798034 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518133 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375409 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527142 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221759 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)
* 03:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444706514 (https://github.com/cluebotng/component-configs/commits/a9515adb6b57ca73316c1c6b8559bc6b49255d7f)
* 03:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183977 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412786 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506484 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 17:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849962 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634738411 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634024663 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615771226 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125894 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788620 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276698067 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510520 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411775 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339656 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225357080 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156826 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-03 ===
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858620 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884839872 (https://github.com/cluebotng/component-configs/commits/cc341cdd381588d285ccbc586ab019ece652d3ce)

=== 2026-05-31 ===
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864789 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858628 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719586315 (https://github.com/cluebotng/component-configs/commits/30600f798367d8fe4124ab5042ddc70122366500)

=== 2026-05-29 ===
* 00:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26610547573 (https://github.com/cluebotng/component-configs/commits/d647b7c087dff8de798d78a98082469fa41e7712)

=== 2026-05-28 ===
* 11:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26572448836 (https://github.com/cluebotng/component-configs/commits/fdea8d6284d14cd9ed835ab73506d0750af62de2)

=== 2026-05-21 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26241143565 (https://github.com/cluebotng/component-configs/commits/76187b1d77a1454297932d833d668fafc6490c10)

=== 2026-05-19 ===
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071084518 (https://github.com/cluebotng/component-configs/commits/6831fe17857c84db1afb1023198b7b11d444f69f)

=== 2026-05-17 ===
* 08:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25985888959 (https://github.com/cluebotng/component-configs/commits/591f7870cefcc3c295b1136fa6afb385738dd863)

=== 2026-05-14 ===
* 17:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda)

=== 2026-05-13 ===
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771881181 (https://github.com/cluebotng/component-configs/commits/d180b98d98a8b5c89115a2080375145455ab4730)

=== 2026-05-12 ===
* 10:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25729275449 (https://github.com/cluebotng/component-configs/commits/ee80e1a674aa4d62ec0736286534130973827ac8)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848224 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)
* 09:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842081 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480610 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)

=== 2026-05-06 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25459711948 (https://github.com/cluebotng/component-configs/commits/3e0d938cc4bbeb5069a3e0b0d864df7eabf78c23)
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451879961 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)
* 16:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25447977391 (https://github.com/cluebotng/component-configs/commits/bd39eba234686d41e4d889def1ed91fc28e8fdf6)

=== 2026-04-24 ===
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24913659688 (https://github.com/cluebotng/component-configs/commits/c5ad64125c64f5db76d9a8141163d380f1c2d4f0)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591292 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)
* 00:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540900579 (https://github.com/cluebotng/component-configs/commits/bfd1f6b6b742e082dbbc587464ca135bfe827765)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007472 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)
* 09:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357383991 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442397 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897956 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249371602 (https://github.com/cluebotng/component-configs/commits/acce8e3abd2da42aef50053d1c1126daec711da1)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248696057 (https://github.com/cluebotng/component-configs/commits/310efa6d212253bd549cddd16d85f4167b26f684)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247621716 (https://github.com/cluebotng/component-configs/commits/2e25667a8eb92011273ca47b217f3546da48cae9)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582931 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093268 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659602695 (https://github.com/cluebotng/component-configs/commits/6f1ef446ee988193ccd8d9c62eb4841267976159)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537319 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959549 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556336 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)
* 16:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383552275 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-14 ===
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358864102 (https://github.com/cluebotng/component-configs/commits/b9059392c8da641c97c97919e1d017bf0251914f)
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733947 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 22:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19347680045 (https://github.com/cluebotng/component-configs/commits/5de2267efa622608c868e4cb66658a4f367aa3b0)
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862269 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)
* 19:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342998586 (https://github.com/cluebotng/component-configs/commits/17a2abf82e9a2076113891e4ae3d5159c048b940)
* 19:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342782944 (https://github.com/cluebotng/component-configs/commits/fed3ef035303fd54334c5a7932d23a43e381492e)
* 18:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342079797 (https://github.com/cluebotng/component-configs/commits/b8c173ced436297aaa9a09b2ef9a4b814d696a30)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341777629 (https://github.com/cluebotng/component-configs/commits/afb04303301910afc2a7692e2e5676540a27c38c)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341671161 (https://github.com/cluebotng/component-configs/commits/84f054bb9402daea356833fbd2d606f4d2b44b48)
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341445508 (https://github.com/cluebotng/component-configs/commits/2a497ee04f7182e6d114d1aafa79a99e29ff6d22)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341306596 (https://github.com/cluebotng/component-configs/commits/6e3dd964163e25f7115b74c7b132316745613729)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341017921 (https://github.com/cluebotng/component-configs/commits/b28d25e5b98fe60c2b85138e7d1a2a52cee03383)
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340976225 (https://github.com/cluebotng/component-configs/commits/8c3a4f743978487c10274534162bb1a7a2ceb8a8)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340717614 (https://github.com/cluebotng/component-configs/commits/a2ac5e2b12d80df59f7f37209ed8943fe471992a)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340563565 (https://github.com/cluebotng/component-configs/commits/a5ea7ec11c7d2a7ced4f29df1f2a742f52dbf777)
* 15:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19337204923 (https://github.com/cluebotng/component-configs/commits/d5247b0e2c72c4426888761d70d73f59ba7362a5)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642882 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270313845 (https://github.com/cluebotng/component-configs/commits/900a30b8983bc80a537330fa4345e7952f2081b4)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264807 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19267535662 (https://github.com/cluebotng/component-configs/commits/eae37aa10505a2ac3c773cdd4f85b7222cdae836)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266909480 (https://github.com/cluebotng/component-configs/commits/bc0dd19078113c4e1cbe90a059fe26932fb70a29)

=== 2025-10-07 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304356335 (https://github.com/cluebotng/component-configs/commits/49f902fb1380f69c5e0156466832b031a9c228ee)

=== 2025-10-06 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272376354 (https://github.com/cluebotng/component-configs/commits/449c7d3c68ade43ce238b9b335182dc50f929511)

=== 2025-10-05 ===
* 06:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255175939 (https://github.com/cluebotng/component-configs/commits/f7e65671f2deb536f52fdf0c1db7ec80709a76da)

=== 2025-10-01 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18153820283 (https://github.com/cluebotng/component-configs/commits/9f9e9abc422c8a74207416afbaf81b72509e91ce)

=== 2025-09-30 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18121221077 (https://github.com/cluebotng/component-configs/commits/11eb3da0c88160bca0ec60e55dadad7e928e40f3)

=== 2025-09-29 ===
* 16:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101421 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101913411 (https://github.com/cluebotng/component-configs/commits/1b6389ae74b8974d6f49591c0abf14a8da974c4b)
* 15:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460176 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217564 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)

=== 2025-09-26 ===
* 06:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030092297 (https://github.com/cluebotng/component-configs/commits/2516589b458034b7c2540c998b29debeb63e82df)

=== 2025-09-25 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801857 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221974 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782304 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677303 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531079 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008148799 (https://github.com/cluebotng/component-configs/commits/0bb2ca770839346853b6a8440927a8afde70c94d)

=== 2025-09-24 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986145339 (https://github.com/cluebotng/component-configs/commits/820291b11edc6256cfa07e9a4a73677df86e52d8)
* 18:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986015781 (https://github.com/cluebotng/component-configs/commits/b0f088ec49a2156a18ad7d78d18640f6e8fe943c)
* 18:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985989347 (https://github.com/cluebotng/component-configs/commits/38b2f2235d143f0522d560a93c04e50e8fa38c77)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985728221 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985446508 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985386789 (https://github.com/cluebotng/component-configs/commits/a1633fc7ecf1bd4a310fcb03d02d0da81212427a)
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333827 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198515 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077117 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820831 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983737370 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 20:30 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17927644810
* 19:02 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925536572


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>qr996efqgo9eb0ozxcfzah7zcniqiwp</sha1>
    </revision>
    <revision>
      <id>2479017</id>
      <parentid>2479000</parentid>
      <timestamp>2026-11-08T11:26:51Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6)</comment>
      <origin>2479017</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="37583" sha1="kqira8heqnr340dr0oism5l9ob0a2ot" xml:space="preserve">=== 2026-11-08 ===
* 11:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6)

=== 2026-11-07 ===
* 12:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373423759 (https://github.com/cluebotng/component-configs/commits/0bc0cef490c8c2afb008ff906644dc7fb977bdb6)
* 12:41 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017679 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372856569 (https://github.com/cluebotng/component-configs/commits/571c4fe0c9c2d2f4afad8b7033bd42575c748d8c)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819815 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)
* 02:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42328251090 (https://github.com/cluebotng/component-configs/commits/9024a7a5b070a1b79f7c9d815774c4afdb9c260f)

=== 2026-11-06 ===
* 18:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42272751673 (https://github.com/cluebotng/component-configs/commits/c17542a324406771d842160c3b55a45fac0d1dae)
* 16:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42250695552 (https://github.com/cluebotng/component-configs/commits/820428026cf62c0b6d4d17e7d385728b4895c891)
* 16:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42249188398 (https://github.com/cluebotng/component-configs/commits/276675d9fde45fb79a69d85827a61fea149951b7)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248468132 (https://github.com/cluebotng/component-configs/commits/973b64f18651ed5b18cf908235604e70028861c8)
* 15:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42240867017 (https://github.com/cluebotng/component-configs/commits/482c4765fb8b7f5b70651b620c2191c412c78234)
* 15:14 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42240676663 (https://github.com/cluebotng/component-configs/commits/4f4442799837e798ff4c86d87a85f93dd57a70e6)
* 14:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42234420334 (https://github.com/cluebotng/component-configs/commits/0c4dbb64c592db6c7c12afbcc69dec8082f6c60b)
* 14:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233833462 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233360609 (https://github.com/cluebotng/component-configs/commits/22a72d8f43f6861bbb9ac592cb9af892dd95f765)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233249218 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42224188055 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42215994580 (https://github.com/cluebotng/component-configs/commits/e3ea44488c33b1bd4dd63a5c9d9c3ef4a58c3406)
* 07:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177719434 (https://github.com/cluebotng/component-configs/commits/66d0b6ee6547ac11df039d6d7e288c9f0acac36b)
* 06:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42177270563 (https://github.com/cluebotng/component-configs/commits/7dfced152f131f82242418b59d55e373d9ad8f77)
* 06:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42172297634 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)
* 03:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42155618016 (https://github.com/cluebotng/component-configs/commits/4552b8960c5335ae823db495604178008b4a4985)

=== 2026-11-05 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099049001 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974709611 (https://github.com/cluebotng/component-configs/commits/5efbb82f84986437a20d11414253c6eb8926c8d2)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300499 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)
* 03:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41961409257 (https://github.com/cluebotng/component-configs/commits/f392ff08eb9768ab93538b5d6ed5cdf5306b9e50)

=== 2026-11-04 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41865423884 (https://github.com/cluebotng/component-configs/commits/86b7159004732c324a71b91e87663d2e1cea32df)
* 03:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208826 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)
* 03:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764161174 (https://github.com/cluebotng/component-configs/commits/efdda3a013777f4e684be77dc6e05704036d4f0a)
* 00:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119449 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6)

=== 2026-11-03 ===
* 18:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693130037 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894047 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194741 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616245 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586263970 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)
* 02:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41567113718 (https://github.com/cluebotng/component-configs/commits/10880256092163e5d6d4d97b1d2a0453e51c1f2c)
* 01:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41561434982 (https://github.com/cluebotng/component-configs/commits/a1b96298b6a3498e8bb51b228a348b2fbd4eed06)

=== 2026-10-31 ===
* 10:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529420 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)
* 09:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41190339868 (https://github.com/cluebotng/component-configs/commits/c258f6030d4d454fbda316e043577097ae61d646)
* 09:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41188726816 (https://github.com/cluebotng/component-configs/commits/0c9a773366a1926ec0b35fc76c3cef9918e56030)
* 08:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41186002297 (https://github.com/cluebotng/component-configs/commits/d2c77aa9fc221ae44769ded27439d0e92a9ec222)

=== 2026-10-30 ===
* 22:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41139595348 (https://github.com/cluebotng/component-configs/commits/902b53df5ae6858a9e61faeae4cf87c175d9e2b1)
* 22:34 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610807 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626137 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41137073295 (https://github.com/cluebotng/component-configs/commits/dbe1aec775bd0f29e13e52e8e5820ade20307178)
* 21:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41128558384 (https://github.com/cluebotng/component-configs/commits/7cb14e452d3768a2f548bca160ca831ab8e67bab)
* 20:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41125890120 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022564 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:40 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40902696917 (https://github.com/cluebotng/component-configs/commits/898b57efb4eeb1bdf98e935d9fe4332693a35aa7)
* 15:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40902351489 (https://github.com/cluebotng/component-configs/commits/a712571c820f2a1ea767bd46ea74112426fb56bc)
* 15:32 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40901778442 (https://github.com/cluebotng/component-configs/commits/abdf069b6a9204f3fe2796741e1beb93d89a5d07)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900617668 (https://github.com/cluebotng/component-configs/commits/342ebf1dca9d2c8f41b88547ed3a017fe1a1a87b)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898272930 (https://github.com/cluebotng/component-configs/commits/9312a5e969e30ca121b0c1dc95882706eb675340)
* 15:10 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40898266178 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 09:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40665456675 (https://github.com/cluebotng/component-configs/commits/cbd8bb224c15c6dacdf93632ef29a11586a0b50b)
* 08:54 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40659788352 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40538846848 (https://github.com/cluebotng/component-configs/commits/3f4a71e8450918f10ab12a9f4b18506226d32f4c)
* 14:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517885090 (https://github.com/cluebotng/component-configs/commits/fbcf68ea18c187fbb8af37ed04a674837d121903)
* 14:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40517784751 (https://github.com/cluebotng/component-configs/commits/3c4486b403207ab117cadd33785b90d078a4f016)
* 13:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40508591039 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 09:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40484261158 (https://github.com/cluebotng/component-configs/commits/698a9bac73dd63611a0686bc1785f80ca9a6683e)
* 06:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40459540079 (https://github.com/cluebotng/component-configs/commits/bc660b38ce5052d918a9c8048fded9d70267c171)
* 05:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40457189661 (https://github.com/cluebotng/component-configs/commits/30587ffc7ef251c2e1a6bcd1d55b7a3b2493d65c)
* 05:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40455798034 (https://github.com/cluebotng/component-configs/commits/80c03e09cf4126cf14f61eae6feded9feafbb264)
* 04:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/40452518133 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375409 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527142 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221759 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)
* 03:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444706514 (https://github.com/cluebotng/component-configs/commits/a9515adb6b57ca73316c1c6b8559bc6b49255d7f)
* 03:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40444183977 (https://github.com/cluebotng/component-configs/commits/368024e9769a6b45b65ce73920e2f880a173e978)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412786 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506484 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)
* 22:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28303337333 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 17:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27635849962 (https://github.com/cluebotng/component-configs/commits/893b9730fafbbd5762e1473e55ad6dbe8cb9db24)
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634738411 (https://github.com/cluebotng/component-configs/commits/a1821e6d783f3a9822825a8325623592bd1754ea)
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27634024663 (https://github.com/cluebotng/component-configs/commits/8a9410bd29ead90dcd183405a7c6ef35c0d898af)
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615771226 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 15:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125894 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 13:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788620 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276698067 (https://github.com/cluebotng/component-configs/commits/8be9293c4b541d74d39482efd21163eb36cda6bd)
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510520 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275411775 (https://github.com/cluebotng/component-configs/commits/869dfb8d1487914e36184a9d1c5aae1e26dbba01)
* 12:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339656 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-06-09 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27225357080 (https://github.com/cluebotng/component-configs/commits/9534cf81437fb2c268eb00e4145978dddbf6322e)

=== 2026-06-08 ===
* 12:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27138156826 (https://github.com/cluebotng/component-configs/commits/4677023bc60821948b76a89b2968d9fa3db267d4)

=== 2026-06-03 ===
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858620 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884839872 (https://github.com/cluebotng/component-configs/commits/cc341cdd381588d285ccbc586ab019ece652d3ce)

=== 2026-05-31 ===
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864789 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858628 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719586315 (https://github.com/cluebotng/component-configs/commits/30600f798367d8fe4124ab5042ddc70122366500)

=== 2026-05-29 ===
* 00:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26610547573 (https://github.com/cluebotng/component-configs/commits/d647b7c087dff8de798d78a98082469fa41e7712)

=== 2026-05-28 ===
* 11:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26572448836 (https://github.com/cluebotng/component-configs/commits/fdea8d6284d14cd9ed835ab73506d0750af62de2)

=== 2026-05-21 ===
* 17:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26241143565 (https://github.com/cluebotng/component-configs/commits/76187b1d77a1454297932d833d668fafc6490c10)

=== 2026-05-19 ===
* 01:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26071084518 (https://github.com/cluebotng/component-configs/commits/6831fe17857c84db1afb1023198b7b11d444f69f)

=== 2026-05-17 ===
* 08:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25985888959 (https://github.com/cluebotng/component-configs/commits/591f7870cefcc3c295b1136fa6afb385738dd863)

=== 2026-05-14 ===
* 17:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda)

=== 2026-05-13 ===
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771881181 (https://github.com/cluebotng/component-configs/commits/d180b98d98a8b5c89115a2080375145455ab4730)

=== 2026-05-12 ===
* 10:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25729275449 (https://github.com/cluebotng/component-configs/commits/ee80e1a674aa4d62ec0736286534130973827ac8)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848224 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)
* 09:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842081 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480610 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)

=== 2026-05-06 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25459711948 (https://github.com/cluebotng/component-configs/commits/3e0d938cc4bbeb5069a3e0b0d864df7eabf78c23)
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451879961 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)
* 16:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25447977391 (https://github.com/cluebotng/component-configs/commits/bd39eba234686d41e4d889def1ed91fc28e8fdf6)

=== 2026-04-24 ===
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24913659688 (https://github.com/cluebotng/component-configs/commits/c5ad64125c64f5db76d9a8141163d380f1c2d4f0)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591292 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)
* 00:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540900579 (https://github.com/cluebotng/component-configs/commits/bfd1f6b6b742e082dbbc587464ca135bfe827765)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007472 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)
* 09:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357383991 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442397 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897956 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249371602 (https://github.com/cluebotng/component-configs/commits/acce8e3abd2da42aef50053d1c1126daec711da1)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248696057 (https://github.com/cluebotng/component-configs/commits/310efa6d212253bd549cddd16d85f4167b26f684)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247621716 (https://github.com/cluebotng/component-configs/commits/2e25667a8eb92011273ca47b217f3546da48cae9)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582931 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093268 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659602695 (https://github.com/cluebotng/component-configs/commits/6f1ef446ee988193ccd8d9c62eb4841267976159)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537319 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959549 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556336 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)
* 16:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383552275 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-14 ===
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358864102 (https://github.com/cluebotng/component-configs/commits/b9059392c8da641c97c97919e1d017bf0251914f)
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733947 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 22:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19347680045 (https://github.com/cluebotng/component-configs/commits/5de2267efa622608c868e4cb66658a4f367aa3b0)
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862269 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)
* 19:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342998586 (https://github.com/cluebotng/component-configs/commits/17a2abf82e9a2076113891e4ae3d5159c048b940)
* 19:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342782944 (https://github.com/cluebotng/component-configs/commits/fed3ef035303fd54334c5a7932d23a43e381492e)
* 18:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342079797 (https://github.com/cluebotng/component-configs/commits/b8c173ced436297aaa9a09b2ef9a4b814d696a30)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341777629 (https://github.com/cluebotng/component-configs/commits/afb04303301910afc2a7692e2e5676540a27c38c)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341671161 (https://github.com/cluebotng/component-configs/commits/84f054bb9402daea356833fbd2d606f4d2b44b48)
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341445508 (https://github.com/cluebotng/component-configs/commits/2a497ee04f7182e6d114d1aafa79a99e29ff6d22)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341306596 (https://github.com/cluebotng/component-configs/commits/6e3dd964163e25f7115b74c7b132316745613729)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341017921 (https://github.com/cluebotng/component-configs/commits/b28d25e5b98fe60c2b85138e7d1a2a52cee03383)
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340976225 (https://github.com/cluebotng/component-configs/commits/8c3a4f743978487c10274534162bb1a7a2ceb8a8)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340717614 (https://github.com/cluebotng/component-configs/commits/a2ac5e2b12d80df59f7f37209ed8943fe471992a)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340563565 (https://github.com/cluebotng/component-configs/commits/a5ea7ec11c7d2a7ced4f29df1f2a742f52dbf777)
* 15:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19337204923 (https://github.com/cluebotng/component-configs/commits/d5247b0e2c72c4426888761d70d73f59ba7362a5)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642882 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270313845 (https://github.com/cluebotng/component-configs/commits/900a30b8983bc80a537330fa4345e7952f2081b4)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264807 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19267535662 (https://github.com/cluebotng/component-configs/commits/eae37aa10505a2ac3c773cdd4f85b7222cdae836)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266909480 (https://github.com/cluebotng/component-configs/commits/bc0dd19078113c4e1cbe90a059fe26932fb70a29)

=== 2025-10-07 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304356335 (https://github.com/cluebotng/component-configs/commits/49f902fb1380f69c5e0156466832b031a9c228ee)

=== 2025-10-06 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272376354 (https://github.com/cluebotng/component-configs/commits/449c7d3c68ade43ce238b9b335182dc50f929511)

=== 2025-10-05 ===
* 06:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255175939 (https://github.com/cluebotng/component-configs/commits/f7e65671f2deb536f52fdf0c1db7ec80709a76da)

=== 2025-10-01 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18153820283 (https://github.com/cluebotng/component-configs/commits/9f9e9abc422c8a74207416afbaf81b72509e91ce)

=== 2025-09-30 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18121221077 (https://github.com/cluebotng/component-configs/commits/11eb3da0c88160bca0ec60e55dadad7e928e40f3)

=== 2025-09-29 ===
* 16:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101421 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101913411 (https://github.com/cluebotng/component-configs/commits/1b6389ae74b8974d6f49591c0abf14a8da974c4b)
* 15:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460176 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217564 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)

=== 2025-09-26 ===
* 06:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030092297 (https://github.com/cluebotng/component-configs/commits/2516589b458034b7c2540c998b29debeb63e82df)

=== 2025-09-25 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801857 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221974 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782304 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677303 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531079 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008148799 (https://github.com/cluebotng/component-configs/commits/0bb2ca770839346853b6a8440927a8afde70c94d)

=== 2025-09-24 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986145339 (https://github.com/cluebotng/component-configs/commits/820291b11edc6256cfa07e9a4a73677df86e52d8)
* 18:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986015781 (https://github.com/cluebotng/component-configs/commits/b0f088ec49a2156a18ad7d78d18640f6e8fe943c)
* 18:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985989347 (https://github.com/cluebotng/component-configs/commits/38b2f2235d143f0522d560a93c04e50e8fa38c77)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985728221 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985446508 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985386789 (https://github.com/cluebotng/component-configs/commits/a1633fc7ecf1bd4a310fcb03d02d0da81212427a)
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333827 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198515 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077117 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820831 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983737370 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 20:30 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17927644810
* 19:02 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925536572


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>kqira8heqnr340dr0oism5l9ob0a2ot</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-editsets/SAL</title>
    <ns>498</ns>
    <id>459255</id>
    <revision>
      <id>2478981</id>
      <parentid>2478864</parentid>
      <timestamp>2026-11-07T12:34:25Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819896 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478981</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="14006" sha1="mebgiqbuq7ma1p7pvshlxrxc87stwux" xml:space="preserve">=== 2026-11-07 ===
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819896 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564952 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249897407 (https://github.com/cluebotng/component-configs/commits/5dc6325f011fee8b3839d39da8c82b95d2894e99)
* 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723422 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833484 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233585957 (https://github.com/cluebotng/component-configs/commits/7bb201355741da2b451f50b4af56b8e395401843)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249040 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188014 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297595 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 20:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099048852 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300317 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208884 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 18:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693129959 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894105 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194793 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616324 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264050 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529431 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610795 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626170 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125890200 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022552 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676356 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757750 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266070 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788301 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591062 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518051 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375417 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527178 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221810 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412759 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506563 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770932 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126038 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788656 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-06-03 ===
* 12:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858556 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)

=== 2026-05-31 ===
* 17:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858597 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)

=== 2026-05-12 ===
* 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848199 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)

=== 2026-05-06 ===
* 17:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25451879892 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640192 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591336 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007498 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357384007 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442420 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898009 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247592403 (https://github.com/cluebotng/component-configs/commits/3a2e7dc89daf45ae2dc369cb6864577d5b18a74b)

=== 2026-04-09 ===
* 18:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093239 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659544147 (https://github.com/cluebotng/component-configs/commits/5a24521cc1a0802f0441809142015e96acf003ad)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959557 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640332 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640332 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556339 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)

=== 2025-11-14 ===
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733943 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862272 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642885 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270276189 (https://github.com/cluebotng/component-configs/commits/040b83debcd71b207ee872095a3d2a1166ccc618)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266879941 (https://github.com/cluebotng/component-configs/commits/7c158c111d99bc1219b8cd64ea79d80a40116c93)

=== 2025-10-29 ===
* 10:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18904984410 (https://github.com/cluebotng/component-configs/commits/6011377571043753ed9ceb9ca00a17bae6883d2a)

=== 2025-09-24 ===
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333851 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198509 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077157 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985011099 (https://github.com/cluebotng/component-configs/commits/f38bf48e73ce94da03cee36fb6cccbe483786e19)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820847 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>mebgiqbuq7ma1p7pvshlxrxc87stwux</sha1>
    </revision>
    <revision>
      <id>2478987</id>
      <parentid>2478981</parentid>
      <timestamp>2026-11-07T12:37:29Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017677 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)</comment>
      <origin>2478987</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="14212" sha1="7wu4k4z0cg1sidjmeh33kzl4pn011rh" xml:space="preserve">=== 2026-11-07 ===
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017677 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819896 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564952 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249897407 (https://github.com/cluebotng/component-configs/commits/5dc6325f011fee8b3839d39da8c82b95d2894e99)
* 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723422 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833484 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233585957 (https://github.com/cluebotng/component-configs/commits/7bb201355741da2b451f50b4af56b8e395401843)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249040 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188014 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297595 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 20:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099048852 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300317 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208884 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 18:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693129959 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894105 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194793 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616324 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264050 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529431 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610795 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626170 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125890200 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022552 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676356 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757750 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266070 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788301 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591062 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518051 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375417 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527178 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221810 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412759 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506563 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770932 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126038 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788656 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-06-03 ===
* 12:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858556 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)

=== 2026-05-31 ===
* 17:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858597 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)

=== 2026-05-12 ===
* 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848199 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)

=== 2026-05-06 ===
* 17:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25451879892 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640192 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591336 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007498 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357384007 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442420 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898009 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247592403 (https://github.com/cluebotng/component-configs/commits/3a2e7dc89daf45ae2dc369cb6864577d5b18a74b)

=== 2026-04-09 ===
* 18:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093239 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659544147 (https://github.com/cluebotng/component-configs/commits/5a24521cc1a0802f0441809142015e96acf003ad)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959557 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640332 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640332 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556339 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)

=== 2025-11-14 ===
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733943 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862272 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642885 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270276189 (https://github.com/cluebotng/component-configs/commits/040b83debcd71b207ee872095a3d2a1166ccc618)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266879941 (https://github.com/cluebotng/component-configs/commits/7c158c111d99bc1219b8cd64ea79d80a40116c93)

=== 2025-10-29 ===
* 10:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18904984410 (https://github.com/cluebotng/component-configs/commits/6011377571043753ed9ceb9ca00a17bae6883d2a)

=== 2025-09-24 ===
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333851 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198509 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077157 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985011099 (https://github.com/cluebotng/component-configs/commits/f38bf48e73ce94da03cee36fb6cccbe483786e19)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820847 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>7wu4k4z0cg1sidjmeh33kzl4pn011rh</sha1>
    </revision>
    <revision>
      <id>2478996</id>
      <parentid>2478987</parentid>
      <timestamp>2026-11-07T12:42:00Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42373303402 (https://github.com/cluebotng/component-configs/commits/0266e661f49a5220ee270323d5bd7eb2617ca4f2)</comment>
      <origin>2478996</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="14421" sha1="kb0shmytvcgihmvs83citzguxvp1yi1" xml:space="preserve">=== 2026-11-07 ===
* 12:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42373303402 (https://github.com/cluebotng/component-configs/commits/0266e661f49a5220ee270323d5bd7eb2617ca4f2)
* 12:37 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373017677 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819896 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564952 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249897407 (https://github.com/cluebotng/component-configs/commits/5dc6325f011fee8b3839d39da8c82b95d2894e99)
* 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723422 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833484 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233585957 (https://github.com/cluebotng/component-configs/commits/7bb201355741da2b451f50b4af56b8e395401843)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233249040 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188014 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297595 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 20:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42099048852 (https://github.com/cluebotng/component-configs/commits/c7e091e072725a2ef015a48313e08c2f2a378bdf)
* 05:30 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41974300317 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208884 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 18:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41693129959 (https://github.com/cluebotng/component-configs/commits/0d19841535066d42c60bc747e02829434852f914)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691894105 (https://github.com/cluebotng/component-configs/commits/4d3b69effc2827247ca575cf060dce20483d3103)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41691194793 (https://github.com/cluebotng/component-configs/commits/7c77c0270dd527e186687a6d6eb0aff11d0ad80c)
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616324 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264050 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529431 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:33 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41138610795 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626170 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 20:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41125890200 (https://github.com/cluebotng/component-configs/commits/8782bd8516e68d4ee63b35a3a86522ed179bceef)

=== 2026-10-29 ===
* 22:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40968022552 (https://github.com/cluebotng/component-configs/commits/cd8fd90b975dd95702ab1dd885b91e8515e197cd)
* 15:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676356 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757750 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898266070 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788301 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591062 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518051 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375417 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527178 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445221810 (https://github.com/cluebotng/component-configs/commits/e6c8388c5259b03b82e9139c4d465a9c0084729f)

=== 2026-07-02 ===
* 21:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412759 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506563 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770932 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285126038 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788656 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-06-03 ===
* 12:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26884858556 (https://github.com/cluebotng/component-configs/commits/1e9d162538309d31e45226e93b81f9d80dae9303)

=== 2026-05-31 ===
* 17:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719858597 (https://github.com/cluebotng/component-configs/commits/6adbf3bfb8326ec2923c1d0a26d44d9325a3a485)

=== 2026-05-12 ===
* 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848199 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)

=== 2026-05-06 ===
* 17:56 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25451879892 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640192 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591336 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007498 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357384007 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442420 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898009 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247592403 (https://github.com/cluebotng/component-configs/commits/3a2e7dc89daf45ae2dc369cb6864577d5b18a74b)

=== 2026-04-09 ===
* 18:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093239 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659544147 (https://github.com/cluebotng/component-configs/commits/5a24521cc1a0802f0441809142015e96acf003ad)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959557 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640332 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640332 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556339 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)

=== 2025-11-14 ===
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733943 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862272 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642885 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270276189 (https://github.com/cluebotng/component-configs/commits/040b83debcd71b207ee872095a3d2a1166ccc618)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266879941 (https://github.com/cluebotng/component-configs/commits/7c158c111d99bc1219b8cd64ea79d80a40116c93)

=== 2025-10-29 ===
* 10:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18904984410 (https://github.com/cluebotng/component-configs/commits/6011377571043753ed9ceb9ca00a17bae6883d2a)

=== 2025-09-24 ===
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333851 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198509 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077157 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985011099 (https://github.com/cluebotng/component-configs/commits/f38bf48e73ce94da03cee36fb6cccbe483786e19)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820847 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>kb0shmytvcgihmvs83citzguxvp1yi1</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebot-syncer/SAL</title>
    <ns>498</ns>
    <id>459263</id>
    <revision>
      <id>2478977</id>
      <parentid>2478863</parentid>
      <timestamp>2026-11-07T12:31:58Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819987 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478977</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="12806" sha1="08d4gausah75wji8px22v9c42yanmcn" xml:space="preserve">=== 2026-11-07 ===
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819987 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250748898 (https://github.com/cluebotng/component-configs/commits/c43bf7a78f253b4a222b93c624e4d7f9a194e8f3)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249636518 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248689057 (https://github.com/cluebotng/component-configs/commits/e265779bb45845ef96baafd067c56b548b4cc7ea)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233853112 (https://github.com/cluebotng/component-configs/commits/20f17c32036d9fb435ac7633a2a983a4927d302d)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833391 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248970 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224187995 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297537 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300416 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208830 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616265 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586264061 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41222920595 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41222230689 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529425 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41138610798 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626149 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41136228472 (https://github.com/cluebotng/component-configs/commits/66a401bd9eb01d9cb9ef3767c627f98334316748)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676310 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757494 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898265777 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788309 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508592543 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518081 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375447 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527408 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445004144 (https://github.com/cluebotng/component-configs/commits/ba4dac835cbaed39f73a3456f14e9543c2fe63ac)

=== 2026-07-11 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 20:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>08d4gausah75wji8px22v9c42yanmcn</sha1>
    </revision>
    <revision>
      <id>2478984</id>
      <parentid>2478977</parentid>
      <timestamp>2026-11-07T12:35:40Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372828537 (https://github.com/cluebotng/component-configs/commits/a2245a8c11578a3bf41b97b386148d20cf6d0688)</comment>
      <origin>2478984</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="13015" sha1="b0kvqi07yka12ddd4q4wbtnmcoc24z8" xml:space="preserve">=== 2026-11-07 ===
* 12:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372828537 (https://github.com/cluebotng/component-configs/commits/a2245a8c11578a3bf41b97b386148d20cf6d0688)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819987 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250748898 (https://github.com/cluebotng/component-configs/commits/c43bf7a78f253b4a222b93c624e4d7f9a194e8f3)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249636518 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248689057 (https://github.com/cluebotng/component-configs/commits/e265779bb45845ef96baafd067c56b548b4cc7ea)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233853112 (https://github.com/cluebotng/component-configs/commits/20f17c32036d9fb435ac7633a2a983a4927d302d)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833391 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248970 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224187995 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297537 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300416 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208830 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616265 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586264061 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41222920595 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41222230689 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529425 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41138610798 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626149 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41136228472 (https://github.com/cluebotng/component-configs/commits/66a401bd9eb01d9cb9ef3767c627f98334316748)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676310 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757494 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898265777 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788309 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508592543 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518081 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375447 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527408 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445004144 (https://github.com/cluebotng/component-configs/commits/ba4dac835cbaed39f73a3456f14e9543c2fe63ac)

=== 2026-07-11 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 20:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>b0kvqi07yka12ddd4q4wbtnmcoc24z8</sha1>
    </revision>
    <revision>
      <id>2478992</id>
      <parentid>2478984</parentid>
      <timestamp>2026-11-07T12:39:07Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373130794 (https://github.com/cluebotng/component-configs/commits/39841188970476424bb18049268ac40290ae604f)</comment>
      <origin>2478992</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="13221" sha1="icvbium04n02qtgzgiwlmj991dbe57n" xml:space="preserve">=== 2026-11-07 ===
* 12:39 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42373130794 (https://github.com/cluebotng/component-configs/commits/39841188970476424bb18049268ac40290ae604f)
* 12:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372828537 (https://github.com/cluebotng/component-configs/commits/a2245a8c11578a3bf41b97b386148d20cf6d0688)
* 12:31 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42372819987 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250748898 (https://github.com/cluebotng/component-configs/commits/c43bf7a78f253b4a222b93c624e4d7f9a194e8f3)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249636518 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248689057 (https://github.com/cluebotng/component-configs/commits/e265779bb45845ef96baafd067c56b548b4cc7ea)
* 14:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233853112 (https://github.com/cluebotng/component-configs/commits/20f17c32036d9fb435ac7633a2a983a4927d302d)
* 14:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233833391 (https://github.com/cluebotng/component-configs/commits/7145135cb4cf221ef0b405732373402fb9d9332d)
* 14:22 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/42233248970 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224187995 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297537 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300416 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:00 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208830 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41632616265 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:51 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41586264061 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 17:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41222920595 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 17:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41222230689 (https://github.com/cluebotng/component-configs/commits/67348a1fbd3bb69bc140d758c10694c2633c1b6d)
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41191529425 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41138610798 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)
* 22:24 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41137626149 (https://github.com/cluebotng/component-configs/commits/99c1ce7bc5ffc90218bb9c2147d679125f717919)
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41136228472 (https://github.com/cluebotng/component-configs/commits/66a401bd9eb01d9cb9ef3767c627f98334316748)

=== 2026-10-29 ===
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40901676310 (https://github.com/cluebotng/component-configs/commits/167ea1cf0b016017fb8c0197ec604ba0461a71c1)
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757494 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898265777 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788309 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508592543 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 05:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518081 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 04:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40447375447 (https://github.com/cluebotng/component-configs/commits/678d113e06e66a0a9093041f8678c92fa82a2d5b)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527408 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)
* 03:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40445004144 (https://github.com/cluebotng/component-configs/commits/ba4dac835cbaed39f73a3456f14e9543c2fe63ac)

=== 2026-07-11 ===
* 20:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 20:02 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:59 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29166081418 (https://github.com/cluebotng/component-configs/commits/d065c0b11340c7f7454de62966fdd2ff5bff5a17)
* 19:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 19:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:09 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 18:06 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162825413 (https://github.com/cluebotng/component-configs/commits/266d284c075a73afd126524c330a6e620c90a977)
* 17:46 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/29162180699 (https://github.com/cluebotng/component-configs/commits/616efbc31e3c889a59147a6740a739bbb33547eb)

=== 2026-07-02 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)
* 21:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51)

=== 2026-07-01 ===
* 13:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-27 ===
* 22:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28303337263 (https://github.com/cluebotng/component-configs/commits/23ce0f851064ca0c4fffa28a0b1510a8ee1dff1c)

=== 2026-06-16 ===
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770917 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 14:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27285125950 (https://github.com/cluebotng/component-configs/commits/3a4f641c7199ec2c34cd294d0baf97b9be997e7b)
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27277788890 (https://github.com/cluebotng/component-configs/commits/39ecf0765b86afbcbd1be02c9f9a5519245ab884)

=== 2026-05-31 ===
* 17:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719582716 (https://github.com/cluebotng/component-configs/commits/b78f0ac1986e68cfb82284ec70f70dd266935962)

=== 2026-05-12 ===
* 08:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723908583 (https://github.com/cluebotng/component-configs/commits/c6402c647a5327dcfe259c406e54dee4150a7c62)

=== 2026-04-10 ===
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442402 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247617592 (https://github.com/cluebotng/component-configs/commits/d4e8e8f9cf7c6848df8726cc0392beac1a400b86)

=== 2026-03-27 ===
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659741719 (https://github.com/cluebotng/component-configs/commits/4dfdf7978d2efacf8a432919fa719c707a917da8)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640339 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383565990 (https://github.com/cluebotng/component-configs/commits/7b73a8f371a4dcd5c1bd4ab710349d08e826f7b1)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642926 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270303362 (https://github.com/cluebotng/component-configs/commits/df94fd299dd1ed07436488c348bad61d859da592)
* 13:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266902429 (https://github.com/cluebotng/component-configs/commits/d31ec1d2c99a6c00020790543c586a10bdb6a8a2)

=== 2025-11-01 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000479779 (https://github.com/cluebotng/component-configs/commits/033e12eae8efed29432ace76735f1edbbe088ed0)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000447906 (https://github.com/cluebotng/component-configs/commits/372f5de91df5a0ffca56b243e1f7fad687f79bcf)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000399780 (https://github.com/cluebotng/component-configs/commits/1a94bd6778a49757134b376395874219208ae1fb)
* 17:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000266719 (https://github.com/cluebotng/component-configs/commits/52c41abedbab9755b5c78d0a036d8181770585c2)
* 17:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19000263881 (https://github.com/cluebotng/component-configs/commits/43f93269012efb5a33c26de345ac33dd270f7025)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217573 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>icvbium04n02qtgzgiwlmj991dbe57n</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.toolforge-functional-runner/SAL</title>
    <ns>498</ns>
    <id>459362</id>
    <revision>
      <id>2478978</id>
      <parentid>2478859</parentid>
      <timestamp>2026-11-07T12:33:45Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819824 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478978</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="8402" sha1="rzdn33rwg0mwrd9jz6xojw03j3a3ml2" xml:space="preserve">=== 2026-11-07 ===
* 12:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819824 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250074402 (https://github.com/cluebotng/component-configs/commits/ab312d3c8f547dc101d3b112f047ff0cb4f3a2cf)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249636608 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233248952 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188056 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297576 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300533 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208828 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616438 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264019 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41191529470 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41138610845 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)

=== 2026-10-29 ===
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757719 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898265808 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788311 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591170 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 04:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518077 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527044 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506415 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804754 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770896 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 12:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510887 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339527 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-05-31 ===
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864809 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)

=== 2026-05-12 ===
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842125 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)

=== 2026-04-10 ===
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442403 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897971 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582903 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093210 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537316 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640333 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383552276 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-11 ===
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19274968338 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9)
* 18:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19274510349 (https://github.com/cluebotng/component-configs/commits/fea93378dd3d44f7cb93bc04bcdeb93272207f7e)
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270643026 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264790 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266868378 (https://github.com/cluebotng/component-configs/commits/1179c63e9137cfff0d1cdc5282eb13246b57ef80)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512910 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>rzdn33rwg0mwrd9jz6xojw03j3a3ml2</sha1>
    </revision>
    <revision>
      <id>2478986</id>
      <parentid>2478978</parentid>
      <timestamp>2026-11-07T12:37:11Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372828492 (https://github.com/cluebotng/component-configs/commits/a2245a8c11578a3bf41b97b386148d20cf6d0688)</comment>
      <origin>2478986</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="8611" sha1="2u0xdifs78wdx10g31ibu56x6tczhuf" xml:space="preserve">=== 2026-11-07 ===
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372828492 (https://github.com/cluebotng/component-configs/commits/a2245a8c11578a3bf41b97b386148d20cf6d0688)
* 12:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819824 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250074402 (https://github.com/cluebotng/component-configs/commits/ab312d3c8f547dc101d3b112f047ff0cb4f3a2cf)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249636608 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42233248952 (https://github.com/cluebotng/component-configs/commits/79c2344352c2d5781360aaadce25e401270c4a47)
* 13:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42224188056 (https://github.com/cluebotng/component-configs/commits/63b10b4f2d7ce4aaf50e16dba7e59d0c415fb566)
* 06:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42172297576 (https://github.com/cluebotng/component-configs/commits/109e459ca496c932b43af3430299a707499c2d1b)

=== 2026-11-05 ===
* 05:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41974300533 (https://github.com/cluebotng/component-configs/commits/62ff28a67ca66036c9cb17267fa8a2cff7a5b641)

=== 2026-11-04 ===
* 03:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208828 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f)

=== 2026-11-03 ===
* 12:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41632616438 (https://github.com/cluebotng/component-configs/commits/f601a120fa49f05bf11c44e48bfeac7b3047b828)
* 05:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41586264019 (https://github.com/cluebotng/component-configs/commits/95f7813d0a27c8f38f34ba200f066a34a2a41e0a)

=== 2026-10-31 ===
* 10:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41191529470 (https://github.com/cluebotng/component-configs/commits/a209b9bfb8365e44534dc5234bf1e079ab62edb0)

=== 2026-10-30 ===
* 22:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41138610845 (https://github.com/cluebotng/component-configs/commits/5dd09b84c6c71125617973814bf1bcde4f34ec3d)

=== 2026-10-29 ===
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40900757719 (https://github.com/cluebotng/component-configs/commits/b49283d3d1ee6fa44d10b759e496ae2c5b230dff)
* 15:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40898265808 (https://github.com/cluebotng/component-configs/commits/2cbcb8f53b16df233c6cf981aa5e005bf42c11ad)

=== 2026-10-28 ===
* 08:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40659788311 (https://github.com/cluebotng/component-configs/commits/ee71eed38729b616c42abe93710f0d6aa01659bd)

=== 2026-10-27 ===
* 13:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40508591170 (https://github.com/cluebotng/component-configs/commits/0fc29f79ec3d781386b9c158ef277442dc02c4a2)
* 04:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40452518077 (https://github.com/cluebotng/component-configs/commits/e6a41bbaf231a25d1cf182623832258b82e34982)
* 03:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/40446527044 (https://github.com/cluebotng/component-configs/commits/12ba566c97fb13fa2d055dd3e9d2df28c1e7d663)

=== 2026-07-01 ===
* 13:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506415 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955)

=== 2026-06-16 ===
* 12:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27616804754 (https://github.com/cluebotng/component-configs/commits/af2bc530f42e1d932c23f8bea9c8d3686c343d10)
* 12:00 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27615770896 (https://github.com/cluebotng/component-configs/commits/38c93594afd6d57d84fde7b3d927c1c73e0a18f9)

=== 2026-06-10 ===
* 12:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27276510887 (https://github.com/cluebotng/component-configs/commits/4442f7413e6335776bd1b8b0a660e20ae1256ae1)
* 12:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27275339527 (https://github.com/cluebotng/component-configs/commits/2d4571e6f74a6269bb7fbd7a03cc1cd1114f0a11)

=== 2026-05-31 ===
* 17:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/26719864809 (https://github.com/cluebotng/component-configs/commits/f9ad39f066688fe2d363bff290d3d8a9e8b5c2a3)

=== 2026-05-12 ===
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842125 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)

=== 2026-04-10 ===
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24250442403 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897971 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582903 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093210 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537316 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640333 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383552276 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-11 ===
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19274968338 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9)
* 18:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19274510349 (https://github.com/cluebotng/component-configs/commits/fea93378dd3d44f7cb93bc04bcdeb93272207f7e)
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270643026 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264790 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266868378 (https://github.com/cluebotng/component-configs/commits/1179c63e9137cfff0d1cdc5282eb13246b57ef80)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512910 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>2u0xdifs78wdx10g31ibu56x6tczhuf</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebot-documentation/SAL</title>
    <ns>498</ns>
    <id>460945</id>
    <revision>
      <id>2478979</id>
      <parentid>2478860</parentid>
      <timestamp>2026-11-07T12:34:06Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819864 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)</comment>
      <origin>2478979</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1125" sha1="sj666bec9k0bt9p2xvelut7xhu1pf3w" xml:space="preserve">=== 2026-11-07 ===
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819864 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564862 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249636361 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723391 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42238039606 (https://github.com/cluebotng/component-configs/commits/3dd7fc0f72de4dcb4aded43e721032ff72f492a0)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>sj666bec9k0bt9p2xvelut7xhu1pf3w</sha1>
    </revision>
    <revision>
      <id>2478990</id>
      <parentid>2478979</parentid>
      <timestamp>2026-11-07T12:37:54Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42373017638 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)</comment>
      <origin>2478990</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1334" sha1="bl8haoi8afyr3xtrkeaiym4b4j2sap3" xml:space="preserve">=== 2026-11-07 ===
* 12:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42373017638 (https://github.com/cluebotng/component-configs/commits/93823d543554cbb8a9c17f2b2550b5eaf018c960)
* 12:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42372819864 (https://github.com/cluebotng/component-configs/commits/e67236d9c095b76ce712a97977daa0796c0aa78c)

=== 2026-11-06 ===
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42250564862 (https://github.com/cluebotng/component-configs/commits/e3de70f78936ff777125f9357795dc9972758e87)
* 16:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42249636361 (https://github.com/cluebotng/component-configs/commits/1eabcb440c70c78d5f010850ca7281a76cb4a0a5)
* 16:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42248723391 (https://github.com/cluebotng/component-configs/commits/2f79137abd5ecc2a7c38afe64d4081cb29378ad6)
* 14:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/42238039606 (https://github.com/cluebotng/component-configs/commits/3dd7fc0f72de4dcb4aded43e721032ff72f492a0)


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>bl8haoi8afyr3xtrkeaiym4b4j2sap3</sha1>
    </revision>
  </page>
  <page>
    <title>User talk:Bancherx</title>
    <ns>3</ns>
    <id>460952</id>
    <revision>
      <id>2479005</id>
      <timestamp>2026-11-07T15:32:29Z</timestamp>
      <contributor>
        <username>StrikerBot</username>
        <id>8475</id>
      </contributor>
      <comment>Welcome to Toolforge!</comment>
      <origin>2479005</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="623" sha1="mh1cikuusve8ql3jo2swsh1wqre27s5" xml:space="preserve">== Welcome to Toolforge! ==
Hello Bancherx, welcome to the Toolforge project!  Your request for access was processed, and you should be able to use ssh to connect to &lt;tt&gt;login.toolforge.org&lt;/tt&gt; and create and manage tools at https://toolsadmin.wikimedia.org/.

Check the [[Help:Toolforge|Toolforge help page]] for tips on using your account.  You can also ask questions in our IRC channel at {{irc|wikimedia-cloud}} or send an e-mail to our mailing list &lt;tt&gt;cloud@lists.wikimedia.org&lt;/tt&gt;.

Thank you, and have fun making Tools! --[[User:StrikerBot|StrikerBot]] ([[User talk:StrikerBot|talk]]) 15:32, 7 November 2026 (UTC)</text>
      <sha1>mh1cikuusve8ql3jo2swsh1wqre27s5</sha1>
    </revision>
  </page>
</mediawiki>