Category:Projects

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
  <siteinfo>
    <sitename>Wikitech</sitename>
    <dbname>labswiki</dbname>
    <base>https://wikitech.wikimedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.47.0-wmf.2</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikitech</namespace>
      <namespace key="5" case="first-letter">Wikitech talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="110" case="first-letter">Obsolete</namespace>
      <namespace key="111" case="first-letter">Obsolete talk</namespace>
      <namespace key="112" case="first-letter">OfficeIT</namespace>
      <namespace key="113" case="first-letter">OfficeIT talk</namespace>
      <namespace key="116" case="first-letter">Tool</namespace>
      <namespace key="117" case="first-letter">Tool talk</namespace>
      <namespace key="498" case="first-letter">Nova Resource</namespace>
      <namespace key="499" case="first-letter">Nova Resource Talk</namespace>
      <namespace key="666" case="first-letter">Heira</namespace>
      <namespace key="667" case="first-letter">Heira Talk</namespace>
      <namespace key="710" case="first-letter">TimedText</namespace>
      <namespace key="711" case="first-letter">TimedText talk</namespace>
      <namespace key="828" case="first-letter">Module</namespace>
      <namespace key="829" case="first-letter">Module talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Category:Projects</title>
    <ns>14</ns>
    <id>1386</id>
    <revision>
      <id>2414319</id>
      <parentid>1771439</parentid>
      <timestamp>2026-05-16T19:59:09Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <origin>2414319</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="221" sha1="k9pzejtyhtaltwlhwl5tmqoxb40hzxs" xml:space="preserve">Overview of [[Portal:Cloud VPS|Cloud VPS]] projects.

To categorize in here, use {{tl|Nova Project Documentation}} on the  Documentation page of your &quot;Nova Resource&quot; namespace page. 

[[Category:Wikimedia infrastructure]]</text>
      <sha1>k9pzejtyhtaltwlhwl5tmqoxb40hzxs</sha1>
    </revision>
  </page>
  <page>
    <title>Deployments</title>
    <ns>0</ns>
    <id>4108</id>
    <revision>
      <id>2414338</id>
      <parentid>2414308</parentid>
      <timestamp>2026-05-17T07:26:05Z</timestamp>
      <contributor>
        <username>ScheduleDeploymentBot</username>
        <id>37566</id>
      </contributor>
      <comment>Add [[gerrit:1288042]] to Monday, May 18 UTC morning backport window</comment>
      <origin>2414338</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="41066" sha1="jmyo2g5f641a3y03ifr95n0jdvw9mxc" xml:space="preserve">{{Navigation MediaWiki deployment}}
This page tracks '''upcoming''' '''deployments''' of software to the [[:m:Special:SiteMatrix|Wikimedia Foundation servers]].

== Getting started ==
Ensure you joined the {{irc|wikimedia-operations}} IRC channel as all deployment-related communications happen there.
If you need help, contact [[:mw:Wikimedia Release Engineering Team|Release Engineering]] on IRC at {{irc|wikimedia-releng}}; and ping Tyler (&lt;code&gt;thcipriani&lt;/code&gt;).

* '''MediaWiki is deployed weekly''' through the [[/Train|Deployment Train]]. Other services follow their own schedule.
* '''Times are pinned to San Francisco''', thus the UTC time changes in March and November per [[:en:Daylight saving time in the United States|DST]].
* '''Prefer regular [[Backport windows]]''' over adding new windows. To request deployment of a config change or backport, add your username and Gerrit URL to one of the backport windows on this page. You must be online in #wikimedia-operations on IRC during your deployment and install [[WikimediaDebug]] ahead of time. The #wikimedia-operations channel requires you to [[:m:IRC/Instructions#Register your nickname, identify, and enforce|register your nickname]] before you can join.
** You can use the '''backport scheduling tool''' to more easily edit this page: &lt;div style=&quot;text-align: center; margin: 1em 0&quot;&gt;{{Clickable button 2|:toollabs:schedule-deployment|Schedule a backport|class=mw-ui-progressive}}&lt;/div&gt;
* Tasks that meet [[/Inclusion criteria|Inclusion criteria]] '''require their own windows''', which includes long-running tasks. '''Schedule more time''' than you think you need to account for delays and set backs, we recommend one hour for most tasks.
**To create or modify a recurring deploy window, send a patchset to [[:gitlab:repos/releng/release/-/blob/main/make-deployment-calendar/deployments-calendar.yaml|deployments-calendar.yaml file]] in &lt;code&gt;repos/releng/release.git&lt;/code&gt;.
**To create an one-off window, simply edit this page accordingly
** '''Announce''' changes to the [[mail:ops|ops mailing list]] ahead of time if you anticipate or are uncertain about noticeable impacts to database load, HTTP caching, or the introduction of new cookies.
** '''Announce''' deployments of major features to the community via [[:m:Tech/News/Next|Tech News]] and/or via other [[:mw:Wikimedia_Product_Guidance/Communication_channels|Product communication channels]].
* '''Something went wrong?''' See [[Incident response]]. Is there a user-impacting problem? Communicate in the {{irc|wikimedia-operations}} IRC channel. If there is a Phabricator task, ensure [[:phab:tag/wikimedia-incident/|#Wikimedia-Incident]] is tagged, and consider setting the [[:mw:Phabricator/Project_management#Priority_levels|Unbreak Now]] priority.
__TOC__
{{anchor|Next Week|Near Term|Near term|Near-term}}{{clear}}
[[Category:Deployment]]
{{Note|content=Subscribe in Google Calendar via &lt;code&gt;wikimedia.org_rudis09ii2mm5fk4hgdjeh1u64@group.calendar.google.com&lt;/code&gt;.&lt;br&gt;This may not include one-off windows. '''If there are differences, then the wiki page is canonical and correct'''.}}

==Week of May 18==


==={{Deployment_day|date=2026-05-17}}===
{{Deployment calendar event card
    |when=2026-05-17 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
==={{Deployment_day|date=2026-05-18}}===
{{Deployment calendar event card
    |when=2026-05-18 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|hubaishan|hubaishan}}
{{deploy|type=config|gerrit=1288042|title=[config] Set Category Collation for arwikisource|status=}} - {{phabricator|T426526}}
{{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-18 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-18 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|Daimona|Daimona (work)}}
{{deploy|type=1.47.0-wmf.2|gerrit=1287895|title=Store uncomputed references delta as null, not 0|status=}} - {{phabricator|T426002}}
* Fixup production data for [[:phab:T426002|T426002]], running query in task description
{{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-18 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-18 08:30 SF
    |length=0.5
    |window=Wikimedia Portals Update
    |who={{ircnick|jan_drewniak|Jan Drewniak}}
    |what=Weekly window for the portals page: https://www.wikipedia.org/
}}
{{Deployment calendar event card
    |when=2026-05-18 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-18 10:00 SF
    |length=0.5
    |window=Wikidata Query Service weekly deploy
    |who={{ircnick|ryankemper|Ryan}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-05-18 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-18 14:00 SF
    |length=2
    |window=Weekly Security deployment window
    |who={{ircnick|alexsanford|Alex}}, {{ircnick|Reedy|Sam}}, {{ircnick|sbassett|Scott}}, {{ircnick|Maryum|Maryum}}, {{ircnick|manfredi|Manfredi}}
    |what=Held deployment window for Security-team related deploys.
}}
{{Deployment calendar event card
    |when=2026-05-18 16:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-18 19:00 SF
    |length=1
    |window=Automatic branching of MediaWiki, extensions, skins, and vendor – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Branch &lt;code&gt;wmf/1.47.0-wmf.3&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-05-18 20:00 SF
    |length=1
    |window=Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/1.47.0-wmf.3&lt;/code&gt; to testwikis
}}
{{Deployment calendar event card
    |when=2026-05-18 21:00 SF
    |length=1
    |window=Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version)
    |who=N/A
    |what=Runs &lt;code&gt;scap clean auto&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-05-18 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-18 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|Amir1|Amir}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}

==={{Deployment_day|date=2026-05-19}}===
{{Deployment calendar event card
    |when=2026-05-19 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-19 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0 Version
    |who={{ircnick|hashar|Antoine}}, {{ircnick|andre|Andre}}
    |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]]
{{DeployOneWeekMini|1.47.0-wmf.2-&gt;1.47.0-wmf.3|1.47.0-wmf.2|1.47.0-wmf.2}}
* group0 to [[mw:MediaWiki_1.47/wmf.3|1.47.0-wmf.3]]
* '''Blockers: {{phabricator|T423912}}'''
}}
{{Deployment calendar event card
    |when=2026-05-19 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-19 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-05-19 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-19 07:00 SF
    |length=0.5
    |window=Test Kitchen UI Deployment Window
    |who=Experimentation Platform Team
    |what=Deployment of Test Kitchen UI (fka MPIC)
}}
{{Deployment calendar event card
    |when=2026-05-19 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-19 08:00 SF
    |length=1
    |window=SRE Collaboration Services office hours
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=Services including Gerrit, Phorge (Phabricator), GitLab
}}
{{Deployment calendar event card
    |when=2026-05-19 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-05-19 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-19 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-19 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-19 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-05-20}}===
{{Deployment calendar event card
    |when=2026-05-20 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-20 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0 Version
    |who={{ircnick|hashar|Antoine}}, {{ircnick|andre|Andre}}
    |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]]
{{DeployOneWeekMini|1.47.0-wmf.3|1.47.0-wmf.2-&gt;1.47.0-wmf.3|1.47.0-wmf.2}}
* group1 to [[mw:MediaWiki_1.47/wmf.3|1.47.0-wmf.3]]
* '''Blockers: {{phabricator|T423912}}'''
}}
{{Deployment calendar event card
    |when=2026-05-20 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-20 04:00 SF
    |length=1
    |window=[[mw:Services|Services]] – [[Citoid]] / [[Zotero]]  
    |who=Marielle ({{ircnick|mvolz}})
    |what=See [[mw:Citoid|Citoid]]
}}
{{Deployment calendar event card
    |when=2026-05-20 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-20 07:00 SF
    |length=1
    |window=Wikifunctions Services UTC Afternoon
    |who=Abstract Wikipedia team (Africa, Europe, Eastern Americas)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-05-20 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-20 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-20 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-20 14:00 SF
    |length=1
    |window=Wikifunctions Services UTC Late
    |who=Abstract Wikipedia team (North and South America)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-05-20 15:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-20 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-20 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|Amir1|Amir}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}
==={{Deployment_day|date=2026-05-21}}===
{{Deployment calendar event card
    |when=2026-05-21 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-21 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0 Version
    |who={{ircnick|hashar|Antoine}}, {{ircnick|andre|Andre}}
    |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]]
{{DeployOneWeekMini|1.47.0-wmf.3|1.47.0-wmf.3|1.47.0-wmf.2-&gt;1.47.0-wmf.3}}
* group2 to [[mw:MediaWiki_1.47/wmf.3|1.47.0-wmf.3]]
* '''Blockers: {{phabricator|T423912}}'''
}}
{{Deployment calendar event card
    |when=2026-05-21 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-21 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-05-21 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-21 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-21 08:00 SF
    |length=1
    |window=Train log triage
    |who={{ircnick|hashar|Antoine}}, {{ircnick|andre|Andre}}
    |what=See [[Heterogeneous deployment/Train deploys#Breakage]]
}}
{{Deployment calendar event card
    |when=2026-05-21 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-05-21 10:00 SF
    |length=1
    |window=Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker)
    |who={{ircnick|bd808}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-05-21 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-21 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|Urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-21 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-21 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-05-22}}===
{{Deployment calendar event card
    |when=2026-05-22 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
{{Deployment calendar event card
    |when=2026-05-22 04:00 SF
    |length=0.5
    |window=GitLab version upgrades
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=GitLab version upgrades
}}
==={{Deployment_day|date=2026-05-23}}===
{{Deployment calendar event card
    |when=2026-05-23 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}

==Week of May 25==


==={{Deployment_day|date=2026-05-24}}===
{{Deployment calendar event card
    |when=2026-05-24 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
==={{Deployment_day|date=2026-05-25}}===
{{Deployment calendar event card
    |when=2026-05-25 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-25 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-25 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-25 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-25 08:30 SF
    |length=0.5
    |window=Wikimedia Portals Update
    |who={{ircnick|jan_drewniak|Jan Drewniak}}
    |what=Weekly window for the portals page: https://www.wikipedia.org/
}}
{{Deployment calendar event card
    |when=2026-05-25 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-25 10:00 SF
    |length=0.5
    |window=Wikidata Query Service weekly deploy
    |who={{ircnick|ryankemper|Ryan}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-05-25 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-25 14:00 SF
    |length=2
    |window=Weekly Security deployment window
    |who={{ircnick|alexsanford|Alex}}, {{ircnick|Reedy|Sam}}, {{ircnick|sbassett|Scott}}, {{ircnick|Maryum|Maryum}}, {{ircnick|manfredi|Manfredi}}
    |what=Held deployment window for Security-team related deploys.
}}
{{Deployment calendar event card
    |when=2026-05-25 16:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-25 19:00 SF
    |length=1
    |window=Automatic branching of MediaWiki, extensions, skins, and vendor – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Branch &lt;code&gt;wmf/1.47.0-wmf.4&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-05-25 20:00 SF
    |length=1
    |window=Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see [[Heterogeneous deployment/Train deploys]]
    |who=N/A
    |what=Deploy &lt;code&gt;wmf/1.47.0-wmf.4&lt;/code&gt; to testwikis
}}
{{Deployment calendar event card
    |when=2026-05-25 21:00 SF
    |length=1
    |window=Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version)
    |who=N/A
    |what=Runs &lt;code&gt;scap clean auto&lt;/code&gt;
}}
{{Deployment calendar event card
    |when=2026-05-25 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-25 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|Amir1|Amir}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}
==={{Deployment_day|date=2026-05-26}}===
{{Deployment calendar event card
    |when=2026-05-26 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-26 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0 Version
    |who={{ircnick|jnuche|Jaime}}, {{ircnick|hashar|Antoine}}
    |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]]
{{DeployOneWeekMini|1.47.0-wmf.3-&gt;1.47.0-wmf.4|1.47.0-wmf.3|1.47.0-wmf.3}}
* group0 to [[mw:MediaWiki_1.47/wmf.4|1.47.0-wmf.4]]
* '''Blockers: {{phabricator|T423913}}'''
}}
{{Deployment calendar event card
    |when=2026-05-26 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-26 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-05-26 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-26 07:00 SF
    |length=0.5
    |window=Test Kitchen UI Deployment Window
    |who=Experimentation Platform Team
    |what=Deployment of Test Kitchen UI (fka MPIC)
}}
{{Deployment calendar event card
    |when=2026-05-26 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-26 08:00 SF
    |length=1
    |window=SRE Collaboration Services office hours
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=Services including Gerrit, Phorge (Phabricator), GitLab
}}
{{Deployment calendar event card
    |when=2026-05-26 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-05-26 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-26 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-26 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-26 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-05-27}}===
{{Deployment calendar event card
    |when=2026-05-27 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-27 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0 Version
    |who={{ircnick|jnuche|Jaime}}, {{ircnick|hashar|Antoine}}
    |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]]
{{DeployOneWeekMini|1.47.0-wmf.4|1.47.0-wmf.3-&gt;1.47.0-wmf.4|1.47.0-wmf.3}}
* group1 to [[mw:MediaWiki_1.47/wmf.4|1.47.0-wmf.4]]
* '''Blockers: {{phabricator|T423913}}'''
}}
{{Deployment calendar event card
    |when=2026-05-27 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-27 04:00 SF
    |length=1
    |window=[[mw:Services|Services]] – [[Citoid]] / [[Zotero]]  
    |who=Marielle ({{ircnick|mvolz}})
    |what=See [[mw:Citoid|Citoid]]
}}
{{Deployment calendar event card
    |when=2026-05-27 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-27 07:00 SF
    |length=1
    |window=Wikifunctions Services UTC Afternoon
    |who=Abstract Wikipedia team (Africa, Europe, Eastern Americas)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-05-27 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-27 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-27 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-27 14:00 SF
    |length=1
    |window=Wikifunctions Services UTC Late
    |who=Abstract Wikipedia team (North and South America)
    |what=Wikifunctions back-end k8s services
}}
{{Deployment calendar event card
    |when=2026-05-27 15:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-27 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-27 23:00 SF
    |length=0.5
    |window=Primary database switchover
    |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|Amir1|Amir}}, {{ircnick|federico3|Federico Ceratto}}
    |what=Held deployment window for database primary masters maintenance
}}
==={{Deployment_day|date=2026-05-28}}===
{{Deployment calendar event card
    |when=2026-05-28 00:00 SF
    |length=1
    |window=[[Backport windows|UTC morning backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-28 01:00 SF
    |length=2
    |window=MediaWiki train - Utc-0 Version
    |who={{ircnick|jnuche|Jaime}}, {{ircnick|hashar|Antoine}}
    |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]]
{{DeployOneWeekMini|1.47.0-wmf.4|1.47.0-wmf.4|1.47.0-wmf.3-&gt;1.47.0-wmf.4}}
* group2 to [[mw:MediaWiki_1.47/wmf.4|1.47.0-wmf.4]]
* '''Blockers: {{phabricator|T423913}}'''
}}
{{Deployment calendar event card
    |when=2026-05-28 03:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-28 05:00 SF
    |length=1
    |window=Mobileapps/RESTBase/Wikifeeds
    |who=Content Transform Team
    |what=Content transform team node services (mobileapps/wikifeeds)
}}
{{Deployment calendar event card
    |when=2026-05-28 06:00 SF
    |length=1
    |window=[[Backport windows|UTC afternoon backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-28 07:30 SF
    |length=0.5
    |window=Test Kitchen Experiment Deployment Window
    |who=Test Kitchen
    |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]].
}}
{{Deployment calendar event card
    |when=2026-05-28 08:00 SF
    |length=1
    |window=Train log triage
    |who={{ircnick|jnuche|Jaime}}, {{ircnick|hashar|Antoine}}
    |what=See [[Heterogeneous deployment/Train deploys#Breakage]]
}}
{{Deployment calendar event card
    |when=2026-05-28 09:00 SF
    |length=1
    |window=[[Puppet request window]]&lt;br/&gt;&lt;small&gt;'''(Max 6 patches)'''&lt;/small&gt;
    |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to Puppet change''
}}
{{Deployment calendar event card
    |when=2026-05-28 10:00 SF
    |length=1
    |window=Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker)
    |who={{ircnick|bd808}}
    |what=...
}}
{{Deployment calendar event card
    |when=2026-05-28 10:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
{{Deployment calendar event card
    |when=2026-05-28 13:00 SF
    |length=1
    |window=[[Backport windows|UTC late backport window]]&lt;br/&gt;&lt;small&gt;'''Your patch may or may not be deployed at the sole discretion of the deployer'''&lt;/small&gt;
    |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}}
    |what={{ircnick|irc-nickname|Requesting Developer}}
* ''Gerrit link to backport or config change''
}}
{{Deployment calendar event card
    |when=2026-05-28 14:00 SF
    |length=1
    |window=Readers deployment window
    |who=Readers
    |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start
}}
{{Deployment calendar event card
    |when=2026-05-28 23:00 SF
    |length=1
    |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early)
    |who=SRE team
    |what=MediaWiki-related infrastructure changes that need a kubernetes deployment.
}}
==={{Deployment_day|date=2026-05-29}}===
{{Deployment calendar event card
    |when=2026-05-29 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}
{{Deployment calendar event card
    |when=2026-05-29 04:00 SF
    |length=0.5
    |window=GitLab version upgrades
    |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}}
    |what=GitLab version upgrades
}}
==={{Deployment_day|date=2026-05-30}}===
{{Deployment calendar event card
    |when=2026-05-30 00:00 SF
    |length=24
    |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken.
    |who=
    |what=No Deploys
}}</text>
      <sha1>jmyo2g5f641a3y03ifr95n0jdvw9mxc</sha1>
    </revision>
  </page>
  <page>
    <title>User:Quiddity</title>
    <ns>2</ns>
    <id>8581</id>
    <revision>
      <id>2414331</id>
      <parentid>2400864</parentid>
      <timestamp>2026-05-16T22:27:32Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>9</comment>
      <origin>2414331</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6480" sha1="bgxvtw9cjo54uz7wpurz5gqc6m9wekh" xml:space="preserve">{{Navigation MediaWiki deployment}}
I am [[w:en:User:Quiddity]] (as volunteer) and [[m:User:Quiddity (WMF)]] (as WMF staff)

== ∞ To-do ∞ ==

; Wiki cleanup
&lt;categorytree mode=pages showcount=on depth=1&gt;Page deletion queue&lt;/categorytree&gt;
&lt;categorytree mode=pages showcount=on depth=1&gt;Pages with issues&lt;/categorytree&gt;
* [[Special:SpecialPages]]
** [[Special:LintErrors]]
* Fix &quot;[https://wikitech.wikimedia.org/w/index.php?sort=last_edit_desc&amp;search=insource%3A%2F%22here%5D%22%2F+-intitle%3A%22goals%22+-intitle%3A%22reports%22+-intitle%3A%22updates%22+-intitle%3A%22archive%22+-intitle%3A%22incidents%22&amp;title=Special%3ASearch&amp;profile=advanced&amp;fulltext=1&amp;ns0=1&amp;ns4=1&amp;ns6=1&amp;ns8=1&amp;ns10=1&amp;ns12=1&amp;ns14=1&amp;ns116=1&amp;ns498=1&amp;ns666=1 here]&quot; as link-label (9 false-positives/user-comments)

== Bookmarks ==

* [[Infographics]] -- gallery
** [[Wikimedia infrastructure#Current infographics]]
** [[Performance/Runbook/diagrams.net conventions]]
* [[Wikitech documentation templates]]
* [https://doc.wikimedia.org/ https://'''doc'''.wikimedia.org/] – [[Doc.wikimedia.org]]
* [https://noc.wikimedia.org/ https://'''noc'''.wikimedia.org/] – [[Noc.wikimedia.org]]
** [https://noc.wikimedia.org/conf/ https://noc.wikimedia.org/'''conf'''/] – configuration files 
*** [https://noc.wikimedia.org/conf/CommonSettings.php.txt '''CommonSettings'''.php.txt]
*** [https://noc.wikimedia.org/conf/InitialiseSettings.php.txt '''InitialiseSettings'''.php.txt]
**** [https://noc.wikimedia.org/wiki.php https://noc.wikimedia.org/'''wiki.php'''] – each wiki's config
** [https://noc.wikimedia.org/db.php https://noc.wikimedia.org/'''db.php'''] – database configuration
* [[mw:Manual:$wgDefaultUserOptions|mw:Manual:$'''wgDefaultUserOptions''']] – other settings
* [https://extloc.toolforge.org https://'''extloc'''.toolforge.org/] – lists of extension-installation-locations, by-wiki or by-extension
* [https://dbtree.wikimedia.org/ https://'''dbtree'''.wikimedia.org/] – [[dbtree.wikimedia.org]]
** [[mw:Manual:Database layout]]
* [https://grafana.wikimedia.org/ https://'''grafana'''.wikimedia.org/] – [[grafana.wikimedia.org]]
* [[stats:|https://'''stats'''.wikimedia.org/]] – [[Analytics/Systems/Wikistats 2]]
* [https://analytics.wikimedia.org https://'''analytics'''.wikimedia.org] – [[Analytics/Systems/Dashiki]] – Dashboards and Data Downloads for Wikimedia Projects
* [https://www.wikimediastatus.net/ https://www.'''wikimediastatus'''.net/] – [[wikimediastatus.net]] (see also [[phab:T202061]])
* [https://icinga.wikimedia.org https://'''icinga'''.wikimedia.org] (p) – [[Icinga]] – host and service monitoring software
* [https://integration.wikimedia.org/ https://'''integration'''.wikimedia.org/] – [[integration.wikimedia.org]] – [[mw:Continuous integration]]
** [https://gerrit.wikimedia.org https://'''gerrit'''.wikimedia.org] – [[Gerrit]] – code review system
*** [https://gerrit.wikimedia.org/r/#/q/project:mediawiki/extensions/UserMerge+status:merged+-owner:%22L10n-bot%22+-owner:%22Libraryupgrader%22+branch:master '''Search gerrit''' without the automated cruft (example)]
** [https://integration.wikimedia.org/ci/ https://integration.wikimedia.org/'''ci'''/] – [[Jenkins]] – continuous integration system
** [https://integration.wikimedia.org/zuul/ https://integration.wikimedia.org/'''zuul'''/] – [[mw:Continuous integration/Zuul|mw:.../Zuul]] – gateway between Gerrit and Jenkins
* [https://toolhub.wikimedia.org/ https://'''toolhub'''.wikimedia.org/] - [[toolhub.wikimedia.org]] - [[m:Toolhub]]
** [https://admin.toolforge.org/tools https://admin.toolforge.org/'''tools'''] – list
** [https://openstack-browser.toolforge.org/project/ https://'''openstack-browser'''.toolforge.org/project/] - list
* [https://codesearch.wmcloud.org/search/ https://'''codesearch'''.wmcloud.org/search/] – Search ALL the code!
** [https://codesearch-beta.wmcloud.org/search/ https://'''codesearch-beta'''.wmcloud.org/search/]
* [[toolforge:global-search|https://'''global-search'''.toolforge.org/]] – Search ALL the wikis!
* [[toolforge:nagf/|https://'''nagf'''.toolforge.org/]] – monitoring graphs
* [[toolforge:replag/|https://'''replag'''.toolforge.org/]] – WMCS Wiki Replicas replication lag
* [[toolforge:tool-db-usage/|https://'''tool-db-usage'''.toolforge.org/]]
* [[toolforge:versions/|https://'''versions'''.toolforge.org/]] – Wikimedia MediaWiki current versions
* [https://federico.kapsi.fi/crstats/core.txt https://federico.kapsi.fi/'''crstats'''/core.txt] &amp; [https://federico.kapsi.fi/crstats/extensions.txt /extensions.txt]
* [[MediaWiki at WMF]]
* [[Reporting a connectivity issue]] -- use [https://wikitech-static.wikimedia.org/wiki/Reporting_a_connectivity_issue wikitech-static version] when sending link to external reports
* [[mw:Help:Locating broken scripts]]
* [[mw:No-JavaScript notes]]
* [[mw:Developers/Maintainers]]
* [[mw:Upstream projects]]
* [[mw:Community metrics]]
* [[mw:MediaWiki history]]
* [[m:IRC/Bots]]
* [[w:en:Wikipedia:Don't worry about performance]]
* [[Help:Toolforge/Toolforge standards committee]] (former member)

== Whimsical lists ==

(For eventual poetry mis-use. Please add words or have a go at combining!) (Because: [[Labs labs labs]] and [[mw:Naming things]]. Play with words, for fun!)

;MediaWiki [languages / packages / services] that are also [common / ambiguous] words
: babel, BagOStuff, banana, blubber, carbon, cite, collection, composer, cucumber, cumin, diamond, druid, echo, flow, fresh, fresnel, ganglia, graph, graphite, Jenkins, kafka, kraken, leaflet, limn, lint, milkshake, nuke, nutcracker, pimple, pivot, pontoon, Prometheus, puppet, quarry, quibble, quiz, ruby, salt, score, skin, sphinx, sentry, squid, subversion, swift, thanks, translate, trebuchet, varnish, zookeeper

;Wikimedia Language-Codes that are also English words
: ace, am, an, arc, as, bar, be, bug, fur, gag, got, ha, haw, he, hi, it, koi, lad, nap, new, no, om, or, pi, rap, simple, so, to, war, yo

;E.g. 
#Kafka Jenkins's flow be diamond it so fresh.
#He be no blubber kraken, he be swift sentry squid, thanks to carbon ganglia.
#Ace composer, Prometheus, echoes thanks to squid zookeeper, as he got fresh banana milkshake.

ʢ´• ᴥ •`ʡ

== Eventualism project ==

* [[User:Quiddity/How it all works]] - halp?


== Subpages ==

{{Special:Prefixindex/User:Quiddity/|hideredirects=1|stripprefix=1}}


== Navboxen ==

{{Navigation Data Platform}}

{{Navigation Wikimedia infrastructure|layout=portal}}</text>
      <sha1>bgxvtw9cjo54uz7wpurz5gqc6m9wekh</sha1>
    </revision>
  </page>
  <page>
    <title>Tool:DrTrigonBot/Migration from TS</title>
    <ns>116</ns>
    <id>12015</id>
    <revision>
      <id>2414313</id>
      <parentid>1761193</parentid>
      <timestamp>2026-05-16T19:50:46Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <comment>incorrect HTML</comment>
      <origin>2414313</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="9624" sha1="het4fbeeoulmqe3nshu9pshhcjczs43" xml:space="preserve">
;→ [[Tool Labs/Migration to eqiad#Phase III (restoring bulk copied tools)]], [[bugzilla:62976]]
;→ migration; check [[Nova Resource:Tools/Help#Configuring mail -- mail forwarding]] and [[bugzilla:58796]] ([[mw:Wikimedia Labs/Tool Labs/Roadmap en#Overview of available features in Tool Labs]])
;→→ migrate TS bots &lt;tt&gt;subster.py&lt;/tt&gt;, &lt;tt&gt;subster_irc.py&lt;/tt&gt; to labs
;→→ should also get a maillist [[Lists.wikimedia.org]], [[meta:Mailing_lists#Create_a_new_list]] in order to redirect bot mails to it, see [[bugzilla:61427]]

== Migration Overview [tools/tool-labs] ==
{| class=&quot;wikitable&quot;
!WMDE Toolserver (TS)
!WMF LabsConsole (labs)
!Description
!Status
|-
|https://toolserver.org/~drtrigon/
|http://tools.wmflabs.org/drtrigonbot/
&lt;tt&gt;/data/project/drtrigonbot/public_html/&lt;/tt&gt;&lt;br&gt;
|webspace root
| style=&quot;background-color:#00FF00&quot; |{{done}}&lt;ref&gt;Display instead of download .log files: http://stackoverflow.com/questions/3935580/htaccess-addtype-plain-text (&lt;tt&gt;/data/project/public_html/drtrigon/DrTrigonBot/.htaccess&lt;/tt&gt;)&lt;/ref&gt; (also for http://tools.wmflabs.org/)
|-
|https://toolserver.org/~drtrigon/cgi-bin/
&lt;br&gt;
* [[:tools:~drtrigon/cgi-bin/panel.py]]
* [[:tools:~drtrigon/cgi-bin/substersim.py]]
* [[:tools:~drtrigon/cgi-bin/sum_cat_disc.py]]
* [[:tools:~drtrigon/cgi-bin/subster_mail_queue.py]]
* [[:tools:~drtrigon/cgi-bin/filter.py]]
* [[:tools:~drtrigon/cgi-bin/xsalt.py]]
** use with e.g.: atom2html.xslt, rss2html.xslt
&lt;br&gt;
|http://tools.wmflabs.org/drtrigonbot/cgi-bin/
&lt;tt&gt;/data/project/drtrigonbot/cgi-bin/&lt;/tt&gt;
* http://tools.wmflabs.org/drtrigonbot/cgi-bin/panel.py
* http://tools.wmflabs.org/drtrigonbot/cgi-bin/substersim.py
* http://tools.wmflabs.org/drtrigonbot/cgi-bin/sum_cat_disc.py
* http://tools.wmflabs.org/drtrigonbot/cgi-bin/subster_mail_queue.py
* http://tools.wmflabs.org/drtrigonbot/cgi-bin/filter.py
* http://tools.wmflabs.org/drtrigonbot/cgi-bin/xsalt.py
&lt;br&gt;
*http://tools.wmflabs.org/drtrigonbot/cgi-bin/info.py
|webspace cgi
| style=&quot;background-color:#00FF00&quot; |{{done}}
|-
|[http://toolserver.org/%7Edrtrigon/cgi-bin/subster_mail_queue.py drtrigon+subster@toolserver.org]
|[[bugzilla:58796]] ([[mw:Wikimedia Labs/Tool Labs/Roadmap en#Overview of available features in Tool Labs]])
|webmail
| style=&quot;background-color:#FF0000&quot; |'''OPEN''' (not available but &quot;planned&quot;)
|-
|http://svn.toolserver.org/svnroot/drtrigon/
https://fisheye.toolserver.org/changelog/drtrigon
|https://phabricator.wikimedia.org/diffusion/PWDT/ &lt;small&gt;[https://github.com/wikimedia/pywikibot-bots-drtrigonbot/ github mirror]&lt;/small&gt;
https://gerrit.wikimedia.org/r/#/projects/pywikibot/bots/drtrigonbot,dashboards/default
|code source repo
| style=&quot;background-color:#00FF00&quot; |{{done}}&lt;ref&gt;http://www.kernel.org/pub/software/scm/git/docs/git-svn.html&lt;/ref&gt;&lt;ref&gt;http://www.pro-linux.de/kurztipps/2/1514/git-und-subversion-gemeinsam-nutzen.html&lt;/ref&gt;&lt;ref&gt;[[mw:Git/New repositories/Requests]]&lt;/ref&gt; (in use)
|-
|https://jira.toolserver.org/browse/DRTRIGON
|https://bugzilla.wikimedia.org/describecomponents.cgi?product=Tool%20Labs%20tools
https://bugzilla.wikimedia.org/describecomponents.cgi?product=Pywikibot
|code issue tracker
| style=&quot;background-color:#00FF00&quot; |{{done}}, see [[bugzilla:55673]]&lt;ref&gt;[[bugzilla:58794]] - Provide upgrade path for jira users&lt;/ref&gt;, [http://tools.wmflabs.org/drtrigonbot/cgi-bin/xsalt.py?url=http%3A%2F%2Ftools.wmflabs.org%2Fdrtrigonbot%2FSearchRequest.xml&amp;xslt=jira2html.xslt jira archive] according to [http://wikimedia.7.x6.nabble.com/What-will-happen-with-the-Toolserver-domain-td5019092.html maillist]
|-
|&lt;tt&gt;~/rewrite/&lt;/tt&gt;
|&lt;tt&gt;/data/project/drtrigonbot/pywikibot-core/&lt;/tt&gt;
|run rewrite bot
| style=&quot;background-color:#00FF00&quot; |{{done}} ([http://tools.wmflabs.org/drtrigonbot/cgi-bin/panel.py status], &lt;tt&gt;[http://tools.wmflabs.org/drtrigonbot/logs/core/script_wui-bot.log script_wui.py]&lt;/tt&gt;)
|-
|&lt;tt&gt;~/pywikipedia/&lt;/tt&gt;
|&lt;tt&gt;/data/project/drtrigonbot/pywikibot-compat/&lt;/tt&gt;
|run trunk bot
| style=&quot;background-color:#FFFF00&quot; |{{done}}&lt;ref&gt;needed software installed; https://bugzilla.wikimedia.org/show_bug.cgi?id=53704&lt;/ref&gt; (&lt;tt&gt;sum_disc.py&lt;/tt&gt;, &lt;tt&gt;catimages.py&lt;/tt&gt; - tests running for &lt;tt&gt;subster.py&lt;/tt&gt;, &lt;tt&gt;subster_irc.py&lt;/tt&gt;)
|-
|[https://wiki.toolserver.org/view/Cron#Cronie_and_Cronietab cronie] + [https://wiki.toolserver.org/view/Job_scheduling queue]
|[[Nova Resource:Tools/Help#Submitting, managing and scheduling  jobs on the grid|crontab + queue]] (grid engine)
|setup cron jobs
| style=&quot;background-color:#00FF00&quot; |{{done}} - [http://tools.wmflabs.org/?status status]
|-
|&lt;tt&gt;/home/drtrigon/&lt;/tt&gt; (&lt;tt&gt;~/&lt;/tt&gt;)
|&lt;tt&gt;/data/project/drtrigonbot/&lt;/tt&gt; (&lt;tt&gt;~/&lt;/tt&gt;)
|setup home tools
| style=&quot;background-color:#00FF00&quot; |{{done}} - [https://phabricator.wikimedia.org/diffusion/PWDT/browse/master/fabfile.py fabfile] (setup/install, backup&lt;ref&gt;According to Damianz #wikimedia-labs 2012-12-28; No backup!!! Do version control and look for the &quot;/home&quot; (and else, e.g. &quot;/data&quot;) backup yourself!&lt;/ref&gt;, update by git, list_large_files, ...)
|-
|[https://wiki.toolserver.org/view/Database_access Database access]
|[[Nova Resource:Tools/Help#Database access]]
|(my)sql
| style=&quot;background-color:#00FF00&quot; |{{done}} (see [http://tools.wmflabs.org/drtrigonbot/cgi-bin/sum_cat_disc.py sum_cat_disc.py])
|-
|https://wiki.toolserver.org/view/Default_license
https://wiki.toolserver.org/view/Toollizenz/draft/en (GPL-compatible)
|http://opensource.org/licenses/GPL-3.0 (may be later LGPL-3.0)&lt;br&gt;some libraries other (e.g. MIT, ...)
|licensing
| style=&quot;background-color:#00FF00&quot; |{{done}} ('free software' might be too strict for some libs)
|-
|https://wiki.toolserver.org/view/User:DrTrigon
|(publish extension page; my git repos, pywikibot also if not done already, ...)
|extension page
| style=&quot;background-color:#FF0000&quot; |'''OPEN''' (open source too strict)
|-
|https://toolserver.org/~drtrigon/doc/
&lt;br&gt;
(containing &lt;tt&gt;DrTrigonBot&lt;/tt&gt;)
|http://tools.wmflabs.org/drtrigonbot/docs/
&lt;tt&gt;/data/project/drtrigonbot/public_html/docs/&lt;/tt&gt;&lt;br&gt;
(like &lt;tt&gt;logs&lt;/tt&gt; containing &lt;tt&gt;compat&lt;/tt&gt;, &lt;tt&gt;core&lt;/tt&gt;, etc.)
|code docs
| style=&quot;background-color:#00FF00&quot; |{{done}} - compat docs generation works and links present
|-
|...
|...
|...
|...
|}

== Migration Overview [bots - obsolete] ==
{| class=&quot;wikitable&quot;
!WMDE Toolserver (TS)
!WMF LabsConsole (labs)
!Description
!Status
|-
|https://toolserver.org/~drtrigon/
|http://bots.wmflabs.org/~drtrigon/
&lt;tt&gt;/data/project/DrTrigonBot/public_html/&lt;/tt&gt;&lt;br&gt;
&lt;tt&gt;/data/project/public_html/drtrigon/&lt;/tt&gt;
|webspace root
|{{done}}&lt;ref&gt;Display instead of download .log files: http://stackoverflow.com/questions/3935580/htaccess-addtype-plain-text (&lt;tt&gt;/data/project/public_html/drtrigon/DrTrigonBot/.htaccess&lt;/tt&gt;)&lt;/ref&gt; (not setup yet)
|-
|https://toolserver.org/~drtrigon/cgi-bin/
|http://bots.wmflabs.org/~drtrigon/cgi-bin/
&lt;tt&gt;/data/project/public_html/drtrigon/cgi-bin/&lt;/tt&gt;
|webspace cgi
|{{done}} (not setup yet)
needs some work, e.g. [http://bots.wmflabs.org/~drtrigon/cgi-bin/panel.py?action=logstat panel.py?action=logstat]
|-
|[http://toolserver.org/%7Edrtrigon/cgi-bin/subster_mail_queue.py drtrigon+subster@toolserver.org]
|([http://bots.wmflabs.org/~drtrigon/cgi-bin/subster_mail_queue.py mail address ???])&lt;ref&gt;According to Damianz #wikimedia-labs 2012-12-28; not possible at the moment, no idea when it will be. But is some kind of planned.&lt;/ref&gt;
|webmail
|'''OPEN''' (not available but &quot;planned&quot;)
|-
|http://svn.toolserver.org/svnroot/drtrigon/
https://fisheye.toolserver.org/changelog/drtrigon
|([[mw:Git/Creating new repositories]])
|code source repo
|'''OPEN''' (use git-svn&lt;ref&gt;http://www.kernel.org/pub/software/scm/git/docs/git-svn.html&lt;/ref&gt;&lt;ref&gt;http://www.pro-linux.de/kurztipps/2/1514/git-und-subversion-gemeinsam-nutzen.html&lt;/ref&gt;)
|-
|https://jira.toolserver.org/browse/DRTRIGON
|(https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs)
|code issue tracker
|'''OPEN''' (other &quot;product&quot;? conversion needed?)
|-
|&lt;tt&gt;~/rewrite/&lt;/tt&gt;
|&lt;tt&gt;/data/project/DrTrigonBot/rewrite/&lt;/tt&gt;
|run rewrite bot
|{{done}} (not tested yet)
|-
|&lt;tt&gt;~/pywikipedia/&lt;/tt&gt;
|&lt;tt&gt;/data/project/DrTrigonBot/pywikipedia/&lt;/tt&gt;
|run trunk bot
|'''OPEN''' (run and test bots)
|-
|[https://wiki.toolserver.org/view/Cron#Cronie_and_Cronietab cronie] + [https://wiki.toolserver.org/view/Job_scheduling queue]
|[[Help:Cron|crontab]]&lt;ref&gt;http://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work&lt;/ref&gt; + [http://linux.die.net/man/1/pgrep pgrep]
(as well as [http://linux.die.net/man/8/anacron anacron], watchdog&lt;ref&gt;http://stackoverflow.com/questions/2366693/run-cron-job-only-if-it-isnt-already-running&lt;/ref&gt;&lt;ref&gt;http://stackoverflow.com/questions/8084142/check-if-rpm-exists-in-bash-script-silently&lt;/ref&gt;, [http://linux.die.net/man/1/monit monit] or &quot;queued&quot; solution&lt;ref&gt;According to Ryan_Lane #wikimedia-labs 2012-12-29; crontab + watchdog is ok. Queue (like SGE) not needed at the moment (enough ressources) but might come in future, e.g. for db access.&lt;/ref&gt;)
|setup cron jobs
|{{done}}
|-
|&lt;tt&gt;~/&lt;/tt&gt;
|&lt;tt&gt;/data/project/DrTrigonBot/&lt;/tt&gt; and &lt;tt&gt;~/&lt;/tt&gt;
|setup home tools
|'''OPEN''' (setup backup&lt;ref&gt;According to Damianz #wikimedia-labs 2012-12-28; No backup!!! Do version control and look for the &quot;/home&quot; (and else, e.g. &quot;/data&quot;) backup yourself!&lt;/ref&gt; and other maintenance tools)
|-
|[https://wiki.toolserver.org/view/Database_access Database access]
|([[Help:Move_your_bot_to_Labs#Set_up_your_bot|January or February 2013]])
|(my)sql
|'''OPEN''' (sql support for tools)
|-
|...
|...
|licensing
|'''OPEN''' (open source too strict)
|-
|...
|...
|...
|...
|}

== References ==
&lt;references/&gt;</text>
      <sha1>het4fbeeoulmqe3nshu9pshhcjczs43</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.wikibugs/SAL</title>
    <ns>498</ns>
    <id>14256</id>
    <revision>
      <id>2414311</id>
      <parentid>2408740</parentid>
      <timestamp>2026-05-16T17:12:34Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~bd808@tools-bastion-14: Restart irc job;bot not in some expected channels (T426501)</comment>
      <origin>2414311</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="68881" sha1="676ggm14knc4ptiygv4h588ufoqm6b6" xml:space="preserve">=== 2026-05-16 ===
* 17:12 wmbot~bd808@tools-bastion-14: Restart irc job;bot not in some expected channels ([[phab:T426501|T426501]])

=== 2026-05-02 ===
* 18:59 wmbot~bd808@tools-bastion-14: Restart of irc to pick up new container ([[phab:T424779|T424779]], [[phab:T421334|T421334]])
* 18:58 wmbot~bd808@tools-bastion-14: Restart of phorge to pick up new container ([[phab:T424779|T424779]], [[phab:T421334|T421334]])
* 18:57 wmbot~bd808@tools-bastion-14: Restart of gitlab to pick up new container ([[phab:T424779|T424779]], [[phab:T421334|T421334]])
* 18:57 wmbot~bd808@tools-bastion-14: Restart of gerrit to pick up new container ([[phab:T424779|T424779]], [[phab:T421334|T421334]])
* 18:54 wmbot~bd808@tools-bastion-14: Hard stop/start of webservice to pick up new container ([[phab:T424779|T424779]], [[phab:T421334|T421334]])
* 18:42 wmbot~bd808@tools-bastion-14: Built new image from git hash {{Gerrit|9a801357}} ([[phab:T424779|T424779]], [[phab:T421334|T421334]])

=== 2026-04-30 ===
* 11:12 wikibugs: Updated channels.yaml to: {{Gerrit|75b8ede37a2b00f4a6d1b230320b983b89391f64f3f30d87d57519c5d314daf0}}

=== 2026-03-20 ===
* 01:21 wmbot~bd808@tools-bastion-14: toolforge jobs restart irc ([[phab:T420663|T420663]])
* 01:14 wmbot~bd808@tools-bastion-14: toolforge jobs restart znc ([[phab:T420663|T420663]])

=== 2026-03-16 ===
* 17:15 wikibugs: Updated channels.yaml to: {{Gerrit|c131d7a4dbc73caa9bf08ed6977d51e50edb3635bf7247a2d93e52b07afdcef2}}
* 17:14 wikibugs: Updated gerrit-channels.yaml to: {{Gerrit|fdd4e3f0a3fc846dc5f2a21cbbd79f9be75773e6197a261769aa7022babb8f2e}}

=== 2026-02-05 ===
* 16:42 wikibugs: Updated channels.yaml to: {{Gerrit|9c1a06cc40414f16ad0d839fc01366b34649da7b7b4761831c09e56a04e6d33c}}

=== 2026-02-04 ===
* 22:41 wikibugs: Updated channels.yaml to: {{Gerrit|1a336492acbc26df1c01342a6f3ccb977d39e06a6b57cb6a06c5844d3dd6ebac}}

=== 2025-11-26 ===
* 23:05 wmbot~bd808@tools-bastion-14: Restarted phorge job to pick up new container ([[phab:T402454|T402454]])
* 23:02 wmbot~bd808@tools-bastion-14: Built new image from git hash {{Gerrit|2cb5431f}}

=== 2025-11-18 ===
* 14:59 taavi: $ toolforge jobs restart irc # to get it to join channels after the bouncer pod was moved to a different node

=== 2025-10-24 ===
* 23:09 wmbot~bd808@tools-bastion-14: Restarted irc, gerrit, and gitlab jobs to pick up new envvars ([[phab:T408139|T408139]])
* 23:04 wmbot~bd808@tools-bastion-14: Set IRC_CHANNELS_FILE, GERRIT_CHANNELS_FILE, and GITLAB_CHANNELS_FILE envvars ([[phab:T408139|T408139]])

=== 2025-10-14 ===
* 10:43 wmbot~taavi@tools-bastion-14: toolforge jobs restart gerrit

=== 2025-09-17 ===
* 08:53 volans: restarted the gerrit job, was not reporting updates again

=== 2025-09-15 ===
* 07:20 volans: restarted the gerrit job, was not reporting updates since last friday

=== 2025-06-09 ===
* 15:21 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|64a8e04d}}

=== 2025-05-25 ===
* 21:01 Reedy: various restarts... [[phab:T395217|T395217]]

=== 2025-05-19 ===
* 08:32 wmbot~dcaro@tools-bastion-13: restarting phorge+gerrit too as they depend on wikibugs #wikibugs had exited but the process did not die
* 08:31 wmbot~dcaro@tools-bastion-13: toolforge webservice restart #wikibugs had exited but the process did not die

=== 2025-04-04 ===
* 10:31 wmbot~lucaswerkmeister@tools-bastion-13: toolforge jobs restart znc; toolforge jobs restart irc # dropped from IRC

=== 2025-03-12 ===
* 17:41 wmbot~bd808@tools-bastion-12: `toolforge jobs restart gitlab` to pick up new GitLab PAT
* 17:33 wmbot~bd808@tools-bastion-12: Rotated GITLAB_TOKEN envvar to use new non-expiring &quot;toolforge-ro&quot; GitLab PAT associated with the https://gitlab.wikimedia.org/pywikibugs account

=== 2025-02-18 ===
* 16:45 wmbot~lucaswerkmeister@tools-bastion-13: toolforge jobs restart znc; toolforge jobs restart irc # dropped from IRC

=== 2025-02-03 ===
* 11:54 andrewbogott: 'webservice restart' and 'toolforge jobs restart znc' and 'toolforge jobs restart irc' because it vanished from IRC

=== 2024-10-26 ===
* 07:33 taavi: restart stuck IRC container

=== 2024-10-21 ===
* 16:36 wmbot~bd808@tools-bastion-12: Restarting gitlab job after reports of missed IRC output and `ValueError(&quot;Chunk too big&quot;)` in logs

=== 2024-10-17 ===
* 22:06 bd808: Bot dropped offline when it lost connection to libera.net but ZNC noticed and recovered without intervention

=== 2024-09-02 ===
* 16:20 wmbot~fnegri@tools-bastion-13: toolforge jobs restart znc
* 16:19 wmbot~fnegri@tools-bastion-13: toolforge jobs restart irc
* 16:18 wmbot~fnegri@tools-bastion-13: toolforge jobs restart irc
* 16:16 wmbot~fnegri@tools-bastion-13: toolforge jobs restart irc

=== 2024-07-16 ===
* 13:17 wmbot~lucaswerkmeister@tools-bastion-13: toolforge jobs restart irc
* 13:15 wmbot~lucaswerkmeister@tools-bastion-13: toolforge jobs restart znc

=== 2024-06-26 ===
* 09:25 wmbot~taavi@tools-bastion-12: toolforge jobs restart phorge

=== 2024-06-25 ===
* 14:09 dhinus: webservice restart; jobs restart irc (lost IRC connection)

=== 2024-05-17 ===
* 22:33 wmbot~bd808@tools-bastion-12: Restarted irc job ([[phab:T364575|T364575]])
* 22:05 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|e6134784}} ([[phab:T364575|T364575]])

=== 2024-05-15 ===
* 00:38 wmbot~bd808@tools-bastion-12: Restart gitlab job to pick up new image
* 00:11 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|f9879fac}}

=== 2024-05-14 ===
* 17:00 wmbot~bd808@tools-bastion-12: Configure GITLAB_EVENTS_URL to point to http://gitlab-webhooks.tool-gitlab-webhooks.svc.tools.local:8000/sse/ to bypass k8s ingress and Toolforge front proxy. ([[phab:T364490|T364490]])

=== 2024-05-13 ===
* 23:21 wmbot~bd808@tools-bastion-12: Restarted gitlab job to pick up fix for [[phab:T364719|T364719]]
* 23:20 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|efd174cd}} ([[phab:T364719|T364719]])
* 21:22 wmbot~bd808@tools-bastion-12: Restarted gerrit, gitlab, irc, and phorge tasks to pick up health-check fixes from MR!41
* 21:14 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|f727d6c1d8}}
* 20:24 wmbot~bd808@tools-bastion-12: Restarted gitlab job to pick up MR owner reporting
* 20:23 wmbot~bd808@tools-bastion-12: Built new image from git hash f7d9ef07.

=== 2024-05-10 ===
* 21:25 wmbot~bd808@tools-bastion-12: Restarted gitlab job to pick up addition MR actions ([[phab:T364615|T364615]])
* 21:23 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|b63637c0}} ([[phab:T364615|T364615]]).
* 17:29 wmbot~bd808@tools-bastion-12: Restarted irc job to enable health check
* 17:27 wmbot~bd808@tools-bastion-12: Restarted phorge job to enable health check
* 17:26 wmbot~bd808@tools-bastion-12: Restarted gerrit job to enable health check
* 17:25 wmbot~bd808@tools-bastion-12: Restarted gitlab job to enable health check ([[phab:T364490|T364490]])
* 17:22 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|35ad598f}} ([[phab:T364490|T364490]]).

=== 2024-05-09 ===
* 15:17 wmbot~bd808@tools-bastion-12: Restarted gitlab task. Nothing logged since 2024-05-09T06:37:49Z ([[phab:T364490|T364490]])

=== 2024-05-08 ===
* 16:45 wmbot~bd808@tools-bastion-12: Restarted gitlab task. Nothing logged since 2024-05-07T20:49:15Z ([[phab:T364490|T364490]])

=== 2024-05-07 ===
* 17:36 wmbot~bd808@tools-bastion-12: Restarted gitlab task. Nothing logged since 2024-05-06T02:01:08Z

=== 2024-05-05 ===
* 10:12 wmbot~bd808@tools-bastion-12: Built new image from git hash {{Gerrit|4a80583b}} ([[phab:T362500|T362500]]).

=== 2024-04-15 ===
* 21:12 wmbot~bd808@tools-bastion-12: Restarted irc task to pick up new container image from  git hash 406e9e18. ([[phab:T360353|T360353]])
* 21:10 wmbot~bd808@tools-bastion-12: Built new image from git hash 406e9e18.

=== 2024-04-12 ===
* 21:34 wmbot~bd808@tools-bastion-12: Restarted phorge task to pick up new container image from  git hash e15756da.
* 21:13 wmbot~bd808@tools-bastion-12: Built new image from git hash e15756da.

=== 2024-04-10 ===
* 07:54 wmbot~taavi@tools-bastion-12: toolforge jobs restart gerrit

=== 2024-04-09 ===
* 17:58 wmbot~bd808@tools-bastion-12: Built new image from git hash 0353f9b0.

=== 2024-04-08 ===
* 16:36 wikibugs: Restarted web, irc, gerrit, and phorge tasks to pick up new image. ([[phab:T361518|T361518]])
* 16:29 wmbot~bd808@tools-bastion-12: Built new image from git hash 0c4ecb64. ([[phab:T361518|T361518]])

=== 2024-04-05 ===
* 13:15 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-04-04 ===
* 14:38 wmbot~samtar@tools-sgebastion-10: Restart all the jobs!

=== 2024-04-03 ===
* 17:58 wmbot~bd808@tools-bastion-12: Built new image and restarted all tasks to pick up Python 3.12 runtime bump
* 10:57 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-04-02 ===
* 12:44 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-04-01 ===
* 21:10 wmbot~bd808@tools-bastion-12: Restart Phorge task to pick up lower backoff limit from {{Gerrit|eabd23c9}}
* 11:13 wmbot~lucaswerkmeister@tools-sgebastion-10: toolforge jobs restart irc

=== 2024-03-31 ===
* 23:20 wmbot~bd808@tools-bastion-12: Updated all to {{Gerrit|f31b4bd}} and buildservice for everything ([[phab:T360860|T360860]])
* 23:16 wikibugs: Updated gerrit-channels.yaml to: {{Gerrit|f31b4bd1456762a012ed0c2059a12832beac3ff0}} channelfilter: fix self._read_from_url call

=== 2024-03-26 ===
* 13:20 wmbot~taavi@tools-bastion-13: toolforge jobs restart irc

=== 2024-03-25 ===
* 12:43 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-03-24 ===
* 21:11 wmbot~bd808@tools-bastion-12: Restarted phoge job to pick up fixes for [[phab:T105625|T105625]] &amp; [[phab:T357875|T357875]]
* 20:53 wmbot~bd808@tools-bastion-12: Updated to f2af9f67; restarted phorge job ([[phab:T359883|T359883]]); restarted gerrit job

=== 2024-03-22 ===
* 17:55 wmbot~bd808@tools-bastion-12: Restarted irc task to pick up {{Gerrit|3069193}} ([[phab:T359982|T359982]], [[phab:T360074|T360074]])
* 17:47 bd808: Installed async-timeout-4.0.3 &amp; redis-5.0.3 in venv as prep for landing MR!15
* 12:51 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-03-21 ===
* 15:50 wmbot~bd808@tools-bastion-12: Restareted irc task. It had dropped from #wikmedia-cloud and not reattached.
* 13:06 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-03-20 ===
* 12:35 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-03-19 ===
* 12:49 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc
* 09:08 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-03-18 ===
* 10:14 wmbot~taavi@tools-bastion-12: toolforge jobs restart irc

=== 2024-03-17 ===
* 11:32 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc

=== 2024-03-16 ===
* 18:40 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc

=== 2024-03-15 ===
* 09:43 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc

=== 2024-03-14 ===
* 10:34 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc

=== 2024-03-13 ===
* 16:21 wmbot~bd808@tools-sgebastion-11: Changed branch protections on https://gitlab.wikimedia.org/toolforge-repos/wikibugs2 to disallow push to main by all. {{Gerrit|fa4fbd54}} was accidentally pushed to main rather than a feature branch.
* 16:18 wmbot~bd808@tools-sgebastion-11: Restared gerrit task to pick up {{Gerrit|fa4fbd54}} fix for [[phab:T360038|T360038]]
* 16:08 wmbot~bd808@tools-sgebastion-11: Restared gerrit task to pick up {{Gerrit|82caa471}} fix for [[phab:T360038|T360038]]
* 15:04 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart gerrit
* 15:03 wmbot~lucaswerkmeister@tools-sgebastion-10: toolforge jobs restart irc
* 12:40 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc
* 11:23 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc

=== 2024-03-12 ===
* 19:59 wmbot~bd808@tools-sgebastion-11: Update to {{Gerrit|a3bef6e}} ([[phab:T128868|T128868]], [[phab:T140881|T140881]], [[phab:T90339|T90339]], [[phab:T281055|T281055]], [[phab:T180293|T180293]])

=== 2024-03-11 ===
* 16:06 wikibugs: Updated channels.yaml to: {{Gerrit|7b84d44eff3dc778078b51c584614422889380a6}} Send all Gerrit projects to wikimedia-releng

=== 2024-03-10 ===
* 15:57 wmbot~bd808@tools-sgebastion-11: Updated to {{Gerrit|7855ab7}} and restarted gerrit &amp; irc jobs ([[phab:T359096|T359096]])
* 10:33 wmbot~taavi@tools-sgebastion-10: toolforge jobs restart irc

=== 2024-03-09 ===
* 12:42 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc, gerrit

=== 2024-03-08 ===
* 13:39 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc, gerrit
* 10:15 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc, gerrit

=== 2024-03-07 ===
* 21:32 wmbot~bd808@tools-sgebastion-11: Restored git webhook handler
* 21:22 wmbot~bd808@tools-sgebastion-11: Updated to {{Gerrit|c42300f}} ([[phab:T359230|T359230]], [[phab:T359202|T359202]], [[phab:T359228|T359228]])
* 20:51 wmbot~bd808@tools-sgebastion-11: Temporarily disable git webhook handler while landing and deploying changes to channel file handling.
* 08:55 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc
* 00:55 wb-test: Updated channels.yaml to: {{Gerrit|7874c409dcef0ecc4bebc22914c540b37fd06574}} gerrit: use wikibugs2.channelfilter.ChannelFilter
* 00:47 wb-test: Updated channels.yaml to: {{Gerrit|87e88d59ab285c1f990579f37a1ceb7def84c5b8}} gerrit: use wikibugs2.channelfilter.ChannelFilter
* 00:42 wb-test: Updated gerrit-channels.yaml to: {{Gerrit|87e88d59ab285c1f990579f37a1ceb7def84c5b8}} gerrit: use wikibugs2.channelfilter.ChannelFilter

=== 2024-03-06 ===
* 18:57 wmbot~bd808@tools-sgebastion-11: Restarting gerrit job; last event logged at 2024-03-06T17:56:59Z ([[phab:T359096|T359096]])
* 18:34 wmbot~bd808@tools-sgebastion-11: Restart phorge task to pick up changes for [[phab:T359145|T359145]] and [[phab:T127506|T127506]]
* 09:46 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc
* 00:00 wb-test: Updated channels.yaml to: {{Gerrit|1909d61ffd9df086a536257524a66f9666687f78}} phorge: Replace `fab` library with local client

=== 2024-03-05 ===
* 18:46 wmbot~bd808@tools-sgebastion-11: Restarting irc job to pick up fix for [[phab:T359097|T359097]]
* 09:24 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart phorge
* 09:17 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart irc

=== 2024-03-04 ===
* 11:04 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart gerrit
* 08:41 taavi: restart irc and gerrit jobs

=== 2024-03-03 ===
* 23:36 wmbot~bd808@tools-sgebastion-11: Restart phorge task to pick up changes for [[phab:T358653|T358653]]
* 15:36 bd808: Replace redis2irc job with znc + irc jobs take 2 ([[phab:T357851|T357851]])
* 15:32 bd808: Replacing legacy wikibugs-phab job with new phorge job ([[phab:T357851|T357851]])
* 15:28 bd808: Replacing legacy grrrrit job with new gerrit job ([[phab:T357851|T357851]])
* 14:49 bd808: Rolled back to redis2irc ([[phab:T357851|T357851]])
* 14:40 bd808: Replace redis2irc job with znc + irc jobs ([[phab:T357851|T357851]])
* 14:00 bd808: Preping for another attempt at migrating to newer codebase ([[phab:T357851|T357851]])

=== 2024-03-02 ===
* 20:59 wmbot~bd808@tools-sgebastion-10: Rebuilt venv for older codebase and restarted all jobs yet again
* 20:42 bd808: Rolled back to pre-refactor code and restarted all jobs
* 20:00 wmbot~bd808@tools-sgebastion-11: Debugging things. I wish this was going more smoothly.
* 19:34 bd808: Building new venv for updated codebase
* 19:26 wmbot~bd808@tools-sgebastion-11: Jobs stopped pending code base switch
* 19:20 wmbot~bd808@tools-sgebastion-11: Deployed znc and service ([[phab:T357729|T357729]])
* 18:37 wmbot~bd808@tools-sgebastion-11: Comment out git pull logic in pull.php while landing and deploying refactor.
* 17:05 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-03-01 ===
* 13:43 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-29 ===
* 12:42 wikibugs: Updated channels.yaml to: {{Gerrit|1d7ee487d6d15c2bd383909a757bb2e2069a6fbe}} Fix typo in the channel name for wikimedia-data-platform

=== 2024-02-28 ===
* 09:14 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart grrrrit

=== 2024-02-27 ===
* 23:33 wmbot~bd808@tools-sgebastion-11: Changed the $HOME/libera git checkout remote origin to https://gitlab.wikimedia.org/toolforge-repos/wikibugs2 ([[phab:T357850|T357850]])
* 20:45 wmbot~bd808@tools-sgebastion-11: Restarted wikibugs-phab job to pick up fixes for [[phab:T1175|T1175]], [[phab:T1176|T1176]], and [[phab:T1177|T1177]]

=== 2024-02-26 ===
* 23:22 wmbot~bd808@tools-sgebastion-11: Restarted wikibugs-phab job. Pod was dead with CreateContainerConfigError state. Last pod event was kubelet pulling the image it runs in.
* 15:07 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart wikibugs-phab
* 12:11 wikibugs: Updated channels.yaml to: {{Gerrit|64ea55e4737bd32a1d8967ddc4fc5d6963a00d19}} Move #data-platform-sre announcements to a dedicated channel
* 10:00 taavi: restart all the wikibugs services

=== 2024-02-25 ===
* 22:05 wmbot~bd808@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-23 ===
* 08:14 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-22 ===
* 11:32 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-21 ===
* 11:57 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc
* 09:46 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-20 ===
* 11:49 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart grrrrit
* 11:05 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-19 ===
* 09:54 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-18 ===
* 09:52 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-17 ===
* 10:39 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc
* 00:02 wmbot~bd808@tools-sgebastion-11: Restarted wikibugs-phab job to pick up fix for [[phab:T199007|T199007]]

=== 2024-02-16 ===
* 12:12 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc
* 11:16 wmbot~taavi@tools-sgebastion-11: toolforge jobs restart redis2irc

=== 2024-02-15 ===
* 11:29 wmbot~lucaswerkmeister@tools-sgebastion-10: toolforge jobs load libera/k8s-jobs.yaml # IRC was down since 9:04 UTC afaict

=== 2024-02-14 ===
* 14:11 wmbot~samtar@tools-sgebastion-10: Restarted all jobs

=== 2024-02-12 ===
* 23:20 wmbot~bd808@tools-sgebastion-11: toolforge jobs restart grrrrit

=== 2024-02-08 ===
* 22:49 bd808: Added greg-g as a co-maintainer
* 22:37 wmbot~bd808@tools-sgebastion-11: Did the `toolforge jobs load libera/k8s-jobs.yaml` full restart
* 22:31 wmbot~bd808@tools-sgebastion-11: Deleted redis2irc-84b494f447-9gvdv pod to try and fix wikibugs irc output after reports by greg-g and TimeStarling

=== 2024-01-10 ===
* 22:07 wikibugs: Updated channels.yaml to: {{Gerrit|4787377098980994d2c1635dccc48cd79da9f446}} channels: Route #openstack-magnum phab tag to -cloud-feed

=== 2023-12-06 ===
* 19:40 wikibugs: Updated channels.yaml to: {{Gerrit|9e1fe76a8be70ee8b6b90c65e9e14fc1481f877e}} remove PAWS from #pywikibot

=== 2023-12-05 ===
* 15:12 wikibugs: Updated channels.yaml to: {{Gerrit|27b1219beab5c5900e7696ccdfd7d37337a715f4}} channels: Add commtech-kanban to commtech channel

=== 2023-11-26 ===
* 14:50 valhallasw: kubectl delete pod grrrrit-567d49c997-m5qzl (pod &quot;grrrrit-567d49c997-m5qzl&quot; deleted); kubectl get pods shows new pod grrrrit-567d49c997-xdvm2 (running 34s)
* 14:48 valhallasw: restarting wikibugs to deploy [[phab:T350778|T350778]] / r975426

=== 2023-10-19 ===
* 12:37 wikibugs: Updated channels.yaml to: {{Gerrit|cd3ece63c7cb70143297fea0343d1fa8e5284c9f}} Remove #wikimedia-quibble (already in #wikimedia-releng)
* 12:34 wikibugs: Updated channels.yaml to: {{Gerrit|cd85fd51baf38429449ea72d88330a1c2d7c249f}} Send Fresnel notifications to Release Engineering

=== 2023-10-09 ===
* 10:58 wikibugs: Updated channels.yaml to: {{Gerrit|ae68917321bf27061fac2dbbfa3f744f50e0a225}} Add wikimedia-external-links to follow linkwatcher

=== 2023-08-31 ===
* 21:08 wikibugs: Updated channels.yaml to: {{Gerrit|eb75d1746a116341e3b7c45514ee31bb489cdb32}} wikistats: add ##RhinosF1

=== 2023-08-11 ===
* 13:00 wikibugs: Updated channels.yaml to: {{Gerrit|e5b0b95a2038396ce31248042141dff2e4a78ce4}} Add Fresh and WikimediaDebug to -releng

=== 2023-07-27 ===
* 15:35 wikibugs: Updated channels.yaml to: {{Gerrit|f7360b75598dfdbceb984305cdba8b76739205ae}} Notify on superset.wmcloud.org

=== 2023-07-25 ===
* 13:05 taavi: move to php 8.2 webservice

=== 2023-07-20 ===
* 22:45 wikibugs: Updated channels.yaml to: {{Gerrit|39a3afa7267b01d24970a552aa5f575298b206b1}} Add #mediawiki-core-bots

=== 2023-07-05 ===
* 14:25 wikibugs: Updated channels.yaml to: {{Gerrit|06dace3d9e6817b50f3a7d190af48d1fdee6962d}} Remove #wikimedia-services

=== 2023-07-01 ===
* 12:06 wikibugs: Updated channels.yaml to: {{Gerrit|2f9a2737781db7d796cca923fa1d186ac40d3a5e}} Add wmde-wikidata-tech to #wikidata-feed

=== 2023-06-19 ===
* 13:42 wikibugs: Updated channels.yaml to: {{Gerrit|e6b7f7783feadc426b2a79f91d5c3c9c761b42a3}} Add 'Data Platform SRE' phab ticket updates to #wikimedia-analytics

=== 2023-05-15 ===
* 18:26 taavi: restart all containers

=== 2023-04-20 ===
* 17:37 wikibugs: Updated channels.yaml to: {{Gerrit|30368f4437745723743bf197bf23faad29c190e4}} Remove services.* from #wikimedia-services
* 17:22 wikibugs: Updated channels.yaml to: {{Gerrit|48c57d89af47960f1181bdd8329ce0bb1f853d02}} Remove blocked-on-services

=== 2023-03-15 ===
* 04:35 legoktm: manually restarted

=== 2023-01-15 ===
* 23:25 wikibugs: Updated channels.yaml to: {{Gerrit|3fc8f04d85b953d989704f66c6ff345b127cd614}} Enable phab and gerrit notifs for #countervandalism

=== 2022-12-31 ===
* 03:21 wikibugs: Updated channels.yaml to: {{Gerrit|352103ccd0bc4ad6355901c8724de306e5ada17c}} channels: Add ##theresnotime-feed

=== 2022-11-21 ===
* 18:29 wikibugs: Updated channels.yaml to: {{Gerrit|c1a96fd7796613e7c5a600147856d9bd9cc25c6d}} Remove unrelated projects from #-fundraising

=== 2022-10-20 ===
* 19:59 wm-bot: &lt;root&gt; restart wikibugs (again)
* 19:12 wm-bot: &lt;root&gt; restart wikibugs

=== 2022-10-10 ===
* 21:08 wm-bot: &lt;root&gt; restart wikibugs
* 20:43 wm-bot: &lt;root&gt; restart wikibugs

=== 2022-10-07 ===
* 17:57 wikibugs: Updated channels.yaml to: {{Gerrit|5c488e7d414102a949371d105d3ec8cb85872976}} channels: route Grid-Engine-to-K8s-Migration to -cloud-feed

=== 2022-07-15 ===
* 13:34 wikibugs: Updated channels.yaml to: {{Gerrit|108bdf1842a176a07e3952355d6843b6f60054d7}} Phab: Match abstract_wiki.* for Abstract channel

=== 2022-06-15 ===
* 18:58 taavi: full shutdown + restart

=== 2022-05-20 ===
* 03:42 wm-bot: &lt;legoktm&gt; restarting IRC bot to lose the tail

=== 2022-05-13 ===
* 04:48 legoktm: restarted wikibugs-phab pod, had stopped reporting

=== 2022-05-06 ===
* 17:06 wikibugs: Updated channels.yaml to: {{Gerrit|783e242631296d112bc37dc9ce1b3112c76a7009}} channels: Community Tech

=== 2022-05-05 ===
* 20:12 wm-bot: &lt;legoktm&gt; Restart webservice so post-merge pull hook works again
* 20:12 wikibugs: Updated channels.yaml to: {{Gerrit|b9b4697dfe8a316d830a85e4c283f90a86b56513}} wikibugs: relay WikiSEO to RhinosF1's channel

=== 2022-03-18 ===
* 16:06 taavi: restarted libera-irc after it flooded out

=== 2022-03-03 ===
* 18:17 wikibugs: Updated channels.yaml to: {{Gerrit|673df01c67a10190f6c2d36e753937684120a182}} channels: route Cloud Services Proposals to cloud-feed
* 08:04 taavi: remove 'taxonomy' cron job on the stretch grid, seems to be intended to update [[mw:Phabricator/Projects]] but broken since early 2021 without anyone complaining

=== 2022-02-28 ===
* 14:05 wikibugs: Updated channels.yaml to: {{Gerrit|1226e1d38c9f1e51e8d99a11ab153b3cb8e7a0c9}} Add #wikimedia-ve for Wikimedia-Venezuela.*

=== 2022-02-25 ===
* 12:16 taavi: fixed auto updated code by restarting webservice to use php 7.4 (instead of 5.6), 5.6 container no longer trusts LE certificates
* 12:15 wikibugs: Updated channels.yaml to: {{Gerrit|e3f9d15ecb3408e823f854ee1891209c24859567}} send openstack stuff to -cloud-feed

=== 2022-01-28 ===
* 14:07 taavi: deploying https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/757403
* 14:07 wikibugs: Updated channels.yaml to: {{Gerrit|4c7bf9f3c273443eba9989f32dd1f563d3378e94}} quibble messages only go to dedicated channel

=== 2021-11-18 ===
* 21:40 wikibugs: Updated channels.yaml to: {{Gerrit|ce25366e1b2a26bd7ee500a1f9fd6358ad311068}} Add #wikimedia-gitlab

=== 2021-10-27 ===
* 18:32 wikibugs: Updated channels.yaml to: {{Gerrit|a353a2cdc026cfc9ca31654366d1dff6d91ea9e6}} Add #wikimedia-quibble

=== 2021-09-16 ===
* 01:07 wikibugs: Updated channels.yaml to: {{Gerrit|260a9bf4ddf30863267b6113dd2c651bfc5d68e4}} channels: Remove 'Vector' from #wikimedia-editing notifs

=== 2021-09-15 ===
* 21:50 legoktm: switched config to use tools-redis.svc.eqiad.wmflabs as redis host and now it seems to work
* 20:03 legoktm: redis2irc is running in a screen because of [[phab:T291129|T291129]]
* 19:33 legoktm: restarted libera-irc to pick up new &quot;In progress&quot; status (didn't actually need to restart libera-phab)
* 19:31 legoktm: restarted libera-phab to pick up new &quot;In progress&quot; status

=== 2021-08-27 ===
* 19:03 wikibugs: Updated channels.yaml to: {{Gerrit|b04aee9d486d7f4f161e1be8ad13179dae52324e}} Fix Wikisource tag names
* 18:44 wikibugs: Updated channels.yaml to: {{Gerrit|27dded8e480c386b4103d4293504eb074e2124a6}} channels: include toolforge milestones

=== 2021-08-06 ===
* 16:09 wikibugs: Updated channels.yaml to: {{Gerrit|b3053ae5642948f4f9828f2db03df5ae884e2617}} Add some extra Wikisource projects to #wikisource

=== 2021-08-03 ===
* 17:46 wikibugs: Updated channels.yaml to: {{Gerrit|1d275d121ea4fab740461b223ddca5c9136ab9d7}} wikibugs: Send Data Persistence events to #wikimedia-data-persistence-feed

=== 2021-07-29 ===
* 18:41 wikibugs: Updated channels.yaml to: {{Gerrit|d7555428ef9264c75b83b097fd78e9918bc8b250}} Send SRE-* projects to `#wikimedia-operations` too

=== 2021-07-15 ===
* 08:26 wikibugs: Updated channels.yaml to: {{Gerrit|8f91a8c4f315dbe38827c0002933589837574d89}} releng: Remove tags RelEng team doesn't own

=== 2021-07-06 ===
* 07:18 wikibugs: Updated channels.yaml to: {{Gerrit|f23263a69fa3084e9a01bbcbef417c914a1ea93c}} Remove ##wmt, project archived
* 07:12 wikibugs: Updated channels.yaml to: {{Gerrit|8e21ef1b7237c7e3ebff38ea999bd78dc16b5329}} Remove #wikimedia-teampractices, project archived

=== 2021-06-25 ===
* 21:50 wikibugs: Updated channels.yaml to: {{Gerrit|45c892ec66874f389a71e57dae009c15430079fd}} Remove #wikimedia-ve, Wikimedia-Venezuela project was archived

=== 2021-06-24 ===
* 08:10 wikibugs: Updated channels.yaml to: {{Gerrit|d7c3b949484194acfe96c70afb23d7e196fb6de5}} Remove debugging channel from a long time ago

=== 2021-06-21 ===
* 21:02 wikibugs: Updated channels.yaml to: {{Gerrit|1afde3d2136dedb493e2c7da34f986ba3a29ca77}} Move #wikimedia-otrs to #wikimedia-vrt, OTRS to Znuny

=== 2021-05-30 ===
* 05:41 legoktm: restarting for irc3 upgrade
* 01:08 legoktm: added &quot;mediawiki&quot; and &quot;ldap/ops&quot; Gerrit groups to ACL for labs/tools/wikibugs2, see wikitech-l email

=== 2021-05-26 ===
* 16:38 wikibugs: Updated channels.yaml to: {{Gerrit|8fe607bc1d6123b999c89b603884825b72e091a7}} Re-add netops to the #wikimedia-traffic channel
* 15:50 wikibugs: Updated channels.yaml to: {{Gerrit|d6a6485cc5ff73c4b2922cd790c273b736580a4c}} Add config for #wikimedia-sre-foundations
* 15:43 wikibugs: Updated channels.yaml to: {{Gerrit|a5f322b13d1bd954cd90e8ce2692c593d06c53fd}} Update for new directory and network move
* 10:22 wikibugs: Updated channels.yaml to: {{Gerrit|dd9d5a6522bc4aa9b9d4150afb316dd957c60981}} Do not crash when tokens are given
* 10:19 wikibugs: Updated channels.yaml to: {{Gerrit|dd9d5a6522bc4aa9b9d4150afb316dd957c60981}} Do not crash when tokens are given
* 02:43 wm-bot: &lt;legoktm&gt; Shutdown freenode version

=== 2021-05-21 ===
* 09:24 wikibugs: Updated channels.yaml to: {{Gerrit|979d87274de573373fa15ff1ae2653c6da6a9978}} Add wikimedia-irc-libera to -ops
* 04:51 wikibugs: Updated channels.yaml to: {{Gerrit|863999e4eb960144b1494232db6de7d0be3a7a6e}} Send #Wikimedia-Small-Projects-User-Group to `#wikimedia-sp`

=== 2021-05-20 ===
* 18:59 wikibugs: Updated channels.yaml to: {{Gerrit|340295c90d8f3ef8fedcba6f1d3969116d2a08f0}} Use #translatewiki on Libera Chat
* 17:37 wikibugs: Updated channels.yaml to: {{Gerrit|eb1a091308916676df4866b623a5d42035230636}} Update -ai to -ml
* 17:24 legoktm: disabled autopull for freenode instance

=== 2021-05-19 ===
* 14:53 legoktm: now running on irc.libera.chat via ~/libera directory

=== 2021-05-14 ===
* 12:20 wikibugs: Updated channels.yaml to: {{Gerrit|e55fede038ce3d79e7208cfd85fa4c7c8ef6886b}} Remove old config that I do not need anymore

=== 2021-05-04 ===
* 18:30 Majavah: restarting wb2-irc, bot timed out from irc

=== 2021-04-05 ===
* 20:17 legoktm: restarted wb2-phab
* 19:07 Majavah: restart phab listener, had crashed due to invalid json message
* 11:29 Majavah: restart irc relay to get it to re-join #wikimedia-releng

=== 2021-03-25 ===
* 20:27 Majavah: restart gerrit listener
* 20:25 bd808: Force deleted stuck wb2-grrrrit job 22577

=== 2021-03-24 ===
* 03:06 wikibugs: Updated channels.yaml to: {{Gerrit|60862cd3720d630dd7874645ee8d9e3cc8a0fd68}} Send &quot;Performance Team (Radar)&quot; to #wikimedia-perf-bots

=== 2021-03-09 ===
* 00:51 wikibugs: Updated channels.yaml to: {{Gerrit|1600f1be3b0432a920fd7258b6cbad4d3c9b2e0f}} Add additional ai-governance phab tag for #wikimedia-ai

=== 2021-03-05 ===
* 21:11 legoktm: restarted wb2-phab process

=== 2021-02-28 ===
* 18:19 legoktm: added Majavah as a maintainer
* 18:17 legoktm: manually stopped all jobs and started them

=== 2021-02-19 ===
* 07:50 wikibugs: Updated channels.yaml to: {{Gerrit|1337ce1c769f00f89e56e81a43a6a65cb82eb308}} Add new Phabricator tasks to the list for #wikimedia-ai

=== 2021-01-07 ===
* 18:10 wikibugs: Updated channels.yaml to: {{Gerrit|cfb1075f65fe0d4b6b896492846891e24cc5dfd9}} Cleanup no longer used projects
* 17:13 wikibugs: Updated channels.yaml to: {{Gerrit|07eb5156c9a035b4937a6269dc386d36d3726d18}} Update name of SRE project in Phab

=== 2020-12-22 ===
* 00:28 wikibugs: Updated channels.yaml to: {{Gerrit|6687ebb2a6dc348504af31c8766820eb529cfbf7}} Notify analytics for schema changes

=== 2020-10-15 ===
* 20:49 legoktm: $ python3 manage.py start_job wb2-phab

=== 2020-09-18 ===
* 16:29 Reedy: restarted due to huge phab lag to irc

=== 2020-09-08 ===
* 12:38 Reedy: restart to pick up https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/625755
* 12:36 wikibugs: Updated channels.yaml to: {{Gerrit|fed690d7bcd92f17f1f4d090d4a05cf35a52a5d7}} Send #jenkins to #wikimedia-releng

=== 2020-09-04 ===
* 14:10 Reedy: rebooted due to laggy irc echoing

=== 2020-08-26 ===
* 01:11 Reedy: restarted wikibugs because of apparent irc split brain (wikibugs and wikibugs____)

=== 2020-06-28 ===
* 06:21 legoktm: restarted wb2-grrrrit

=== 2020-06-17 ===
* 17:02 wikibugs: Updated channels.yaml to: {{Gerrit|25325355b20a2e2453238d78ddea8d0cd18c8774}} Update zppixbot project name

=== 2020-06-04 ===
* 10:00 wm-bot: &lt;maurelio&gt; Restarting wb2-grrrrit, still not parsing gerrit feed
* 09:43 wm-bot: &lt;maurelio&gt; Restarting wikibugs, not listening to gerrit

=== 2020-04-09 ===
* 18:29 wikibugs: Updated channels.yaml to: {{Gerrit|10c446c88bbebfe5153b16f489ab8d60e81d8c96}} Partially revert &quot;Drop the #wikimedia-interactive channel, no longer used&quot;
* 18:04 wikibugs: Updated channels.yaml to: {{Gerrit|f3ca868d48da550c5882dd07f3008e78e6093ce3}} Drop the #wikimedia-interactive channel, no longer used
* 17:25 wikibugs: Updated channels.yaml to: {{Gerrit|4f74eadc0b86a0fc7c0ed502651518446b6c6a60}} More Product Infrastructure repos

=== 2020-04-04 ===
* 14:27 Reedy: rebooted

=== 2020-03-12 ===
* 01:09 wikibugs: Updated channels.yaml to: {{Gerrit|60932adfb079ee7c3443fbb2f31cf17c817731df}} Add other items RelEng cares about

=== 2020-03-01 ===
* 15:55 wikibugs: Updated channels.yaml to: {{Gerrit|7fc88d6228d00ad5e773c8c4b0065e2471240a03}} Add User-revi to the channels list

=== 2020-02-29 ===
* 23:04 wm-bot: &lt;root&gt; Migrated to 2020 Kubernetes cluster

=== 2020-02-24 ===
* 18:18 wikibugs: Updated channels.yaml to: {{Gerrit|f109caf1cedea816385a1278dd23631d38733d60}} Move RelEng's interest in other's work out

=== 2020-02-11 ===
* 17:32 Reedy: reboot because it wasn't reporting from phab

=== 2019-12-31 ===
* 15:25 valhallasw`cloud: Tried to restart wb2-phab, but now getting an asyncio AttributeError: 'NoneType' object has no attribute 'call_exception_handler'
* 15:22 valhallasw`cloud: temporarily deployed https://gerrit.wikimedia.org/r/#/c/labs/tools/wikibugs2/+/561294/ while waiting for review

=== 2019-12-12 ===
* 13:03 Reedy: restart to fix double posting to irc

=== 2019-10-31 ===
* 17:04 bd808: Restarted bot at request of paladox via irc

=== 2019-10-12 ===
* 15:43 valhallasw`cloud: qdel'ed wb2-grrrrit and restarted with ~/py35-stretch/bin/python manage.py start_jobs

=== 2019-10-11 ===
* 17:30 wikibugs: Updated channels.yaml to: {{Gerrit|f51498e134a76fe72b944e1db7c5f6f38dd16f44}} Update tags for #mediawiki-i18n

=== 2019-10-09 ===
* 15:49 phamhi: restarted wikibugs since it had stopped reporting

=== 2019-09-28 ===
* 18:49 wikibugs: Updated channels.yaml to: {{Gerrit|cbff02ce5e3ed77807d1753a0add44b7b550a720}} Add User-RhinosF1 project alerts for channel ##RhinosF1

=== 2019-09-26 ===
* 21:56 wikibugs: Updated channels.yaml to: {{Gerrit|304a5e24d3b69bddb38cadc08a3c9a0a024e91de}} remove ##Zppix-Wikipedia from ZppixBot notifications

=== 2019-09-08 ===
* 18:54 bd808: Restarting service by following https://www.mediawiki.org/wiki/Wikibugs#Restarting_wikibugs

=== 2019-08-23 ===
* 20:52 wikibugs: Updated channels.yaml to: {{Gerrit|ff7f5fed938e91a41c91200c105f00dc6b3142f1}} Reading-Infrastructure-Team Phab boards have new names

=== 2019-07-17 ===
* 04:50 legoktm: restarted wb2-phab, got stuck?

=== 2019-06-05 ===
* 19:50 wikibugs: Updated channels.yaml to: {{Gerrit|445187388f45f0a9dd7d038cdfe41a5990e373a8}} Add #wikimedia-codehealth channel

=== 2019-05-20 ===
* 05:04 wikibugs: Updated channels.yaml to: {{Gerrit|c51d8f138e1c9cccb1e0b37f44b253ba146116c1}} Remove Google-Code-in-2018 Phabricator board from -dev-africa
* 05:02 legoktm: manually pulled 6dc85ed..c51d8f1 because of a local conflict in manage.py

=== 2019-04-10 ===
* 12:48 arturo: restart wikibugs following docs at https://www.mediawiki.org/wiki/Wikibugs#Restarting_wikibugs

=== 2019-03-15 ===
* 11:21 arturo: run `tools.wikibugs@tools-sgebastion-07:~/wikibugs2$ python3 manage.py start_jobs` as described in https://www.mediawiki.org/wiki/Wikibugswith after deleting the grid jobs

=== 2019-03-06 ===
* 19:37 valhallasw`cloud: wb2-grrrrit deployed on stretch, all happy!
* 19:33 valhallasw`cloud: wb2-phab deployed on stretch
* 19:24 valhallasw`cloud: wb2-irc deployed on stretch, seems to work.
* 19:16 valhallasw`cloud: Taxonomy crontab deployed on stretch
* 19:11 wikibugs: Updated channels.yaml to: {{Gerrit|8c2212415d8280a5b5d4ca16a48f8a5527a160d2}} Bump bs4 to 4.7.1

=== 2019-02-20 ===
* 18:13 wikibugs: Updated channels.yaml to: {{Gerrit|43339cbca935d094fa81c5cb641805b00192e13f}} Remove Quarry from #wikimedia-research

=== 2019-02-17 ===
* 07:41 wikibugs: Updated channels.yaml to: {{Gerrit|62469f2db86d26c599400a55b9a7642ef95ce8d9}} Update for Acme-chief project rename

=== 2019-01-27 ===
* 23:58 wikibugs: Updated channels.yaml to: {{Gerrit|21d84d38c4990405bde8381859fff54507e26dd4}} Send &quot;Zuul&quot; phabricator project updates to #wikimedia-releng

=== 2018-11-15 ===
* 20:20 wikibugs: Updated channels.yaml to: {{Gerrit|13c6f92210f849c0c1b5ec701f9a20b2d663ef04}} Merge &quot;Add maps items to Infrastructure team channel&quot;

=== 2018-11-14 ===
* 19:35 wikibugs: Updated channels.yaml to: {{Gerrit|cf8f688c9a6b3662fe0444547611d74e3ebfe876}} Add MobileFrontend board to -dev &amp; other boards to -dev-africa

=== 2018-10-31 ===
* 01:44 gtirloni: moved gerrit updates to #wikimedia-cloud-feed

=== 2018-10-26 ===
* 00:40 gtirloni: restarted jobs ([[phab:T208009|T208009]])

=== 2018-10-15 ===
* 20:06 wikibugs: Updated channels.yaml to: {{Gerrit|e8b9a96950503ad8bf906d318633b9ddcd8187b1}} channels: Route certcentral tasks to -traffic

=== 2018-10-12 ===
* 17:46 wikibugs: Updated channels.yaml to: {{Gerrit|3a91169fd100b09dd7619b39bcc966e9c3f56f9f}} Fix phab board name for #wikimedia-dev-africa

=== 2018-10-06 ===
* 06:00 wikibugs: Updated channels.yaml to: {{Gerrit|88c0a4fb868be28e9bdf37afdaa970f0d5ab61d7}} Add some tags that might get added on their own, without Scoring-platform-team

=== 2018-10-04 ===
* 19:06 zhuyifei1999_: qmod -rj wb2-phab

=== 2018-09-30 ===
* 16:00 wikibugs: Updated channels.yaml to: {{Gerrit|1242fb010fe0926e5ab0efdedf1e42c4c4ee91ca}} Add Pywikibot-core phab board to #wikimedia-dev-africa

=== 2018-09-21 ===
* 16:58 wikibugs: Updated channels.yaml to: {{Gerrit|59937c1b4d3c31cd15c802831117b1a4c43c6b35}} Add GCI 2018 work-board to #wikimedia-dev-africa

=== 2018-08-27 ===
* 22:00 legoktm: wb2-grrrrit died, restarted manually

=== 2018-08-21 ===
* 13:12 Hauskatze: Restarted wb2-irc job (bot quit due to excess flood)
* 13:03 wikibugs: Updated channels.yaml to: {{Gerrit|66fb9a4cf98864c1a5c226c0fd14229498c7fe83}} Merge branch 'master' of https://gerrit.wikimedia.org/r/labs/tools/wikibugs2

=== 2018-08-15 ===
* 20:57 Hauskatze: restarted gerrit listener (wb2-grrrit) job using documentation at mediawiki.org [job was not running, fabric not used ] {{!}} [[phab:T202027|T202027]]
* 03:38 wikibugs_: Updated channels.yaml to: {{Gerrit|34ba6181df6241b4bdab192f3a83037a4c19e8d6}} Merge branch 'master' of https://gerrit.wikimedia.org/r/labs/tools/wikibugs2

=== 2018-08-10 ===
* 21:51 wikibugs_: Updated channels.yaml to: {{Gerrit|181a9422666bbb5d8ad2c25253909aabe44dddec}} Merge branch 'master' of https://gerrit.wikimedia.org/r/labs/tools/wikibugs2

=== 2018-07-02 ===
* 10:37 valhallasw`cloud: merged ignore-communitytechbot-patch and restarted

=== 2018-06-25 ===
* 17:59 Krenair: restarting wb2-irc as it got killed for excess flood

=== 2018-06-13 ===
* 06:01 wikibugs: Updated channels.yaml to: {{Gerrit|a7eaa3105adc56e4e923a6af879aa3a2db3394ca}} Example Reading-Infrastructure-Team projects

=== 2018-06-12 ===
* 18:22 wikibugs: Updated channels.yaml to: {{Gerrit|7b71ab65e663a9f2b186d695e6bf94902beb319e}} Add #wikimedia-ri-ci with relevant repos and Phab project

=== 2018-06-06 ===
* 18:37 Reedy: qdel current jobs

=== 2018-05-17 ===
* 15:06 legoktm: restarted wb2-phab job

=== 2018-04-12 ===
* 01:50 wikibugs: Updated channels.yaml to: {{Gerrit|3b1b66271bd97799a723e4e1367c4586f3f9d3e5}} Notify #mediawiki-parsoid for more project bugs

=== 2018-03-20 ===
* 15:35 wikibugs: Updated channels.yaml to: {{Gerrit|d038766effad436a63321f9bf90012dad8fbd7b6}} Changing the channel for wikimedia-de related messages

=== 2018-02-24 ===
* 01:27 wikibugs: Updated channels.yaml to: {{Gerrit|7083c324b227fa9cea91dd0f63aeef6c26444ca2}} Merge #wikimedia-anti-harassment-tools into #wikimedia-commtech

=== 2018-01-19 ===
* 20:35 wikibugs: Updated channels.yaml to: {{Gerrit|e85e5b9d9cd2b574d72bf2292b19f8a256f888fd}} Follow-up: I forgot Phabricator's sub-projects
* 19:16 wikibugs: Updated channels.yaml to: {{Gerrit|82f35a133ffe502920845f25a616e3e64de66be2}} Remove #-devtools, merge into #-releng

=== 2018-01-15 ===
* 17:13 d3r1ck: Move IRC channel from #wiki-dev-africa to #wikimedia-dev-africa (proper namespace) for the Africa Wikimedia Developer Project
* 17:12 wikibugs: Updated channels.yaml to: {{Gerrit|558223821cfe3fb742eed65d1fdf3112ce9506a7}} Merge &quot;add #wikimedia-ve to wikibugs&quot;
* 17:11 wikibugs: Updated channels.yaml to: {{Gerrit|dfa7acb7be6172bbbcf1e902f13e2a05603aefa2}} Migration from #wiki-dev-africa to #wikimedia-dev-africa

=== 2018-01-02 ===
* 11:28 legoktm: restarted everything because it was being weird

=== 2017-12-14 ===
* 17:27 wikibugs: Updated channels.yaml to: {{Gerrit|54e2d754583aa29c5fd3d79fad64fae407c78924}} If its not working break it more.

=== 2017-12-13 ===
* 22:04 wikibugs: Updated channels.yaml to: {{Gerrit|09f9fc29c13bc37bd7d6ed3a19c04269c45f2f39}} I promise I understand how to write code and change configs... it may take me 430 patchsets first.
* 21:52 wikibugs: Updated channels.yaml to: {{Gerrit|60c37074c081c2cb39915cb41bcd70607c633084}} Cleanup, Merjin made me do this
* 21:40 wikibugs: Updated channels.yaml to: {{Gerrit|0a8c097f134c17a045e51bf6a36ee3d3b46d80c6}} Merge &quot;releng: Cleanup&quot;

=== 2017-10-24 ===
* 17:55 wikibugs: Updated channels.yaml to: {{Gerrit|dcdb429afd431a1f221c4a038305b1f83056bcfe}} Move wmcs-related notices to #wikimedia-cloud-feed

=== 2017-09-24 ===
* 05:32 legoktm: legoktm: Deployed {{Gerrit|7f85b1857fbd887b53523ab4a312673b31db0ba4}} Fix casing of Libraryupgrader wb2-grrrrit

=== 2017-09-13 ===
* 18:55 valhallasw`cloud: restarting wb2-grrrrit to pull in {{Gerrit|I1dce797340788ce12fda27102abdbb7f97849107}} / [[phab:T174772|T174772]]

=== 2017-08-07 ===
* 16:51 valhallasw`cloud: restarted webservice on grid engine -- the git pull exec does not work on kubernetes

=== 2017-03-30 ===
* 20:43 chasemp: manual jsub commands per https://www.mediawiki.org/wiki/Wikibugs
* 20:38 chasemp: tools.wikibugs@tools-bastion-03:~$ webservice restart

=== 2016-11-17 ===
* {{SAL entry|1=18:29 valhallasw`cloud: killed stray wikibugs on tools-exec-1404}}

=== 2016-11-15 ===
* {{SAL entry|1=18:57 legoktm: restarted wikibugs}}

=== 2016-11-01 ===
* 20:59 legoktm: manually restarting duplicate wb2-phab listener

=== 2016-10-26 ===
* 01:31 wikibugs: Updated channels.yaml to: 06b5d789dc5bb3aa059733595efbcfc50d380d61 Report WMDE team boards to wmde tech chan

=== 2016-10-19 ===
* 17:32 wikibugs: Updated channels.yaml to: 56bdfd42cd03c3cb7322e415f7c5db90e07ce60c Update wikimedia-services mapping

=== 2016-10-07 ===
* 21:40 Krenair: manually shut down and restarted wikibugs, it had split into two separate bots for some reason

=== 2016-10-06 ===
* 22:48 wikibugs: Updated channels.yaml to: e23d40f0cf15b1387847ed9f5972b7e0662d8803 Change project Project-Creators to Project-Admins

=== 2016-09-01 ===
* 01:05 wikibugs: Updated channels.yaml to: 9315b267bc6dea02ee1a10bca37b0bc562d9e514 New naming scheme for Edit-Review-Improvements projects

=== 2016-08-31 ===
* 19:11 legoktm: started redis2irc and wikibugs jobs

=== 2016-07-29 ===
* 05:13 ircnotifier: legoktm: Deployed 79d62890d3d4f56a3f69bad4349795bae6746e50 Ignore @Phabricator_maintenance and clean up code wb2-irc

=== 2016-07-28 ===
* 02:44 Danny_B: started the irc part
* 02:41 Danny_B: killed the irc part for a bit to suppress the spam from batch edit

=== 2016-07-09 ===
* 05:55 wikibugs: Updated channels.yaml to: 837a3228cd3a841566bcaf89c1cd71a7682d7518 Fix the project name for xTools

=== 2016-06-01 ===
* 09:55 wikibugs: Updated channels.yaml to: 9da4fc07d28f33fb74207a78b96fe57068ef6d46 Invite wikibugs to #wikimedia-databases

=== 2016-05-25 ===
* 18:17 ircnotifier: valhallasw: Deployed 6b863811ff4a2ce9230eabce141f802854cd33f7 Merge &quot;add hostname to userinfo&quot; wb2-irc

=== 2016-05-23 ===
* 11:34 valhallasw`cloud: temporarily offline for mass edit by Danny_B 

=== 2016-05-09 ===
* 09:19 ircnotifier: legoktm: Deployed 1715d33c22249a3f1dde01d6eec929b3ad6abb8c Column changes are now &quot;core:columns&quot; wb2-phab

=== 2016-05-08 ===
* 14:13 valhallasw`cloud: adding Danny_B as maintainer so he can stop/restart wikibugs while doing mass edits

=== 2016-05-06 ===
* 19:47 valhallasw`cloud: temporarily offline for Danny_B's batch job
* 09:12 wikibugs: Updated channels.yaml to: 904ce69d7f515ea9b47f621686f612a04ae94cc2 Send WMDE-Design to #wikimedia-de-tech

=== 2016-05-05 ===
* 21:37 wikibugs: Updated channels.yaml to: 6e0a2d140f923140c73b0dd5a356816a1cb47aa5 Add wildcard for Collaboration team (Collab-Team(-.*)?)

=== 2016-04-22 ===
* 00:38 wikibugs: Updated channels.yaml to: 98b78de2f209713741863c9dbf5a14eba7164ccd Update for renamed xTools project

=== 2016-04-21 ===
* 18:04 wikibugs: Updated channels.yaml to: 68d22345224b2e64698527a65b21c1a9cba9f84f Added #wikimedia-interactive

=== 2016-04-15 ===
* 16:40 wikibugs: Updated channels.yaml to: 63a166edb4d8218e1edf42346b5b8a90e4b6d647 Merge &quot;Revert &quot;Log hackathon tasks to #wmhack&quot;&quot;

=== 2016-04-11 ===
* 17:58 ircnotifier: valhallasw: Deployed 170e3ace519867782ecb709eb095059b063d1cd1 Merge &quot;Try to fix display of colours appearing directly before numbers&quot; wb2-irc
* 17:57 wikibugs: Updated channels.yaml to: 4acf9e002ad00a8af3553833f99b754bfc0e189c Merge &quot;Add #wikimedia-ai channel&quot;

=== 2016-04-08 ===
* 22:51 wikibugs: Updated channels.yaml to: 2fc4b94daebe62f8e5e8712d753fabb5878e418a Echo was renamed to Notifications

=== 2016-04-01 ===
* 09:54 wikibugs: Updated channels.yaml to: a1754bac96c8fcb14532b808d5ca9db15e8f3c25 Merge &quot;Log hackathon tasks to #wmhack&quot;

=== 2016-03-29 ===
* 18:53 wikibugs: Updated channels.yaml to: ad14e0d1a0a07b775d014e6f8f1edaf145349116 Add two Collaboration team boards

=== 2016-03-28 ===
* 02:30 wikibugs: Updated channels.yaml to: fcffc4d11cddf0e11b5353bf50a9c36f4d989090 Send Community-Wishlist-Survey stuff to #wikimedia-commtech

=== 2016-02-18 ===
* 18:58 ircnotifier: legoktm: Deployed 8e84aa72b250ef18421c47ace7b4422e949c5837 Ignore comments posted to Phabricator by Stashbot wb2-irc

=== 2016-02-16 ===
* 16:30 wikibugs: Updated channels.yaml to: f4951fed10578ac63d80a6ec95e0927aaa90411e Remove ContentTranslations bugs from #mediawiki-i18n

=== 2016-02-11 ===
* 02:16 wikibugs: Updated channels.yaml to: 164d65c02aa22ec6e53f05ec74c35dfd58d11c24 -releng and -devtools changes

=== 2016-02-08 ===
* 22:10 wikibugs: Updated channels.yaml to: 2dd0d574c0e2bfcd5285493664f884f2ddc54b99 Send Education-* to wikimedia-ed

=== 2016-01-07 ===
* 20:53 wikibugs: Updated channels.yaml to: db26b7db94db89a49fac63df54d0189cf39ffc90 Send Labs* to `#wikimedia-labs`

=== 2015-12-21 ===
* 18:31 valhallasw`cloud: and restarted with fab start-jobs. Welcome back, wikibugs.
* 18:30 valhallasw`cloud: ah, there are SGE processes running. OK, killing those as well. 
* 18:28 valhallasw`cloud: what's even weirder is that it starts both wikibugs.py and redis2irc.py, which are two distinct SGE jobs. Uuh?
* 18:27 valhallasw`cloud: yet it respawns! What on earth. Again from 208.80.155.186, and killed again.
* 18:26 valhallasw`cloud: killed wikibugs manually, no SGE in sight.
* 18:24 valhallasw`cloud: using `listlogins` in nickserv, we find one running on 208.80.155.186 (-1409), one on 208.80.155.145 (-1405, just restarted)
* 18:20 valhallasw`cloud: duplicate wikibugs, trying qmod -rj

=== 2015-12-07 ===
* 20:39 valhallasw`cloud: wb2-irc thinks it's connected but messages don't actually get out to IRC. Restarting.

=== 2015-12-03 ===
* 23:11 wikibugs: Updated channels.yaml to: 74f9c1e0e07d47abc0ca706040faaf90b1ea585d Add PAWS to pywikibot and labs channels

=== 2015-11-04 ===
* 16:44 wikibugs: Updated channels.yaml to: 9a7f239ec5c34604d9e48901fc28e997ea53a5e4 Add #Testing-Initiative-2015 to -releng
* 04:31 wikibugs: Updated channels.yaml to: 775987cc6b7998d7495fcae652546ab2df0d1d6a Send all User-* projects to /dev/null

=== 2015-10-28 ===
* 18:28 wikibugs: Updated channels.yaml to: e5e90fdb7faaa2b992321b1facd2799ae25d61e7 send Mailing lists tickets to #wikimedia-mailman

=== 2015-10-21 ===
* 18:32 wikibugs: Updated channels.yaml to: b4e285f9673929b8547902be466e04d903d3237d Add WMDE-Analytics-Engineering to #wikimedia-de-tech

=== 2015-10-07 ===
* 16:57 wikibugs: Updated channels.yaml to: c78efa6e621b316c26fca060661f59558e8bafa5 Merge &quot;Also exclude TCB-Team- from #wikimedia-fundraising&quot;

=== 2015-10-06 ===
* 00:34 wikibugs: Updated channels.yaml to: 9da0a4809b8d990d2a87d465868d8a8c8fd549b1 Send Beta-Cluster-Infrastructure to #wikimedia-releng

=== 2015-09-27 ===
* 19:34 wikibugs: Updated channels.yaml to: d2f4a855aa8e5f6a9d06e43c35abcc9448f57b73 Add MediaWiki-Codesniffer to -releng

=== 2015-09-24 ===
* 17:41 wikibugs: Updated channels.yaml to: bf1ac0ad9fd2f358aeb62335516c6ca4304b649d Add #MediaWiki-Releasing to -releng
* 16:40 wikibugs: Updated channels.yaml to: eca76c2669d6b0d308b2457b146dbef7f5f91d26 Add #releng-epics to -releng

=== 2015-09-23 ===
* 14:02 wikibugs: Updated channels.yaml to: 06f2a7d0a89baac1a1255c8e2524afa9654e09c1 Send all Community-Tech-* traffic to #wikimedia-commtech

=== 2015-09-22 ===
* 22:54 wikibugs: Updated channels.yaml to: 40a3bfed7706047eb5df3b7f36fda29c54cec3fc Pywikibot-Flow → #wikimedia-collaboration

=== 2015-09-19 ===
* 05:02 wikibugs: Updated channels.yaml to: 4d6ce23148a4fa57c84139465beeb84361ebb6de Add releng-(.*) to catch all releng planning tags

=== 2015-09-18 ===
* 09:20 wikibugs: Updated channels.yaml to: 6a3566594f734c84b9e7d85600ae39b33ab366de releng is now Release-Engineering-Team
* 03:11 wikibugs: Updated channels.yaml to: f9dab0c9be84689bc24046982ef22e22d45402b7 OTRS → #wikimedia-otrs

=== 2015-09-16 ===
* 17:27 ircnotifier: legoktm: Deployed bcce439fda97c0a91b6ef983221f336a3da0cf99 Wait at least 1 second before pushing into redis wb2-phab
* 15:04 wikibugs: Updated channels.yaml to: 08ac39ff3184c179434bb9f36187adcea5ea8f24 Remove ECT and old/dead projects from -devtools

=== 2015-09-07 ===
* 11:07 wikibugs: Updated channels.yaml to: 04c06838cc50d916c6cc11b20776a62b8b5fbdc1 Report ArticlePlaceholder to #wikidata-feed

=== 2015-09-04 ===
* 20:56 wikibugs: Updated channels.yaml to: 0a914ec9da79d3e0b8a6dcbe825a2ac54eb03446 add notifs for #wikimedia-ios room
* 03:13 wikibugs: Updated channels.yaml to: d18a53d6498e5faa77a03b35261f9d26fd51766d Add #Differential for -releng

=== 2015-09-03 ===
* 16:26 wikibugs: Updated channels.yaml to: 6790b1bed753260168db20bb78dcfa5726be2aec Deprecate gitblit, and migrate gerrit
* 05:46 ircnotifier: legoktm: Deployed 1564da8bd2a53f9899e93497f03ba13e4a6b734f Forrestbot → ReleaseTaggerBot wb2-irc

=== 2015-09-02 ===
* 00:03 wikibugs: Updated channels.yaml to: 5f5fbb9243566dd512f1b1bf65ed60e5e08d6e92 Naming is hard

=== 2015-08-28 ===
* 05:14 wikibugs: Updated channels.yaml to: 30a3e422993291ab995a487ae1d4bbc2e7cd4013 Send Community-Tech traffic to #wikimedia-commtech

=== 2015-08-25 ===
* 18:04 wikibugs: Updated channels.yaml to: a541227fe36479f99a74f8d56586bcf4b8f55108 Send Collaboration-Team(-.*)? to #wikimedia-collaboration

=== 2015-08-19 ===
* 08:25 wikibugs: Updated channels.yaml to: b03ba14e0809cf29d6fad807c931b7b9bafb0b2f Add RelEng-Admin, CI-Config, Scap3 plus reorder

=== 2015-08-04 ===
* 17:48 wikibugs: Updated channels.yaml to: 5b039d4ec16094a553be431e4d94f0bf880cfa47 Send GlobalRename changes to #wikimedia-rename

=== 2015-07-30 ===
* 21:11 wikibugs: Updated channels.yaml to: f638b92139d824b52c8e37284b4e5bedf07cf52c Filter WMDE- out of #wikimedia-fundraising

=== 2015-07-28 ===
* 20:59 ircnotifier: legoktm: Deployed 680c8aad81158a3ddb1c4018233c07729c163cc0 Don't notify if multiple ignored actions were triggered wb2-phab

=== July 2 ===
* 18:21 wikibugs: Updated channels.yaml to: ac57db111909071aa63faf61ff9a2a1fee1c693f xtools moved to wikimedia-xtools Change-Id: Ie84324e718d8025f4a1381f36d9ff5f4e9c5848d
* 17:52 valhallasw`cloud: restarting wikibugs using fab to get verbose logs &amp; logrotate back
* 17:33 ircnotifier: valhallasw: Deployed 0f163852ed56e50bbfeb53377a0913570ee21fea Merge &quot;Revert &quot;Use NOTICE instead of PRIVMSG&quot;&quot; wb2-irc
* 17:29 ircnotifier: valhallasw: Deployed bd6cbfabe79cc254c9525f136ae981ef20479c1e Merge &quot;Use NOTICE instead of PRIVMSG&quot; wb2-irc

=== June 15 ===
* 20:13 wikibugs: Updated channels.yaml to: 317ea9408296ac9c0e0b8cfe3b9fe1952ac57f04 Change #wikidata to #wikidata-feed

=== June 10 ===
* 11:38 wikibugs: Updated channels.yaml to: a8b1cb73fc9c7f07e8d8329a5fac09f4974a2c5d Add 3 probjects to #wikimedia-de-tech

=== June 9 ===
* 17:21 wikibugs: Updated channels.yaml to: fb7b824b7b6310ecbe957d07d66eaa4b1dbb8e6e Move ResourceLoader and Performance-Team #wikimedia-perf

=== June 5 ===
* 20:59 ircnotifier: legoktm: Deployed 82b0b9f487ece85a40595b80f3f690554743e472 Ignore Forrestbot wb2-phab, wb2-irc

=== May 25 ===
* 08:18 ircnotifier: valhallasw: Deployed 82b0b9f487ece85a40595b80f3f690554743e472 Ignore Forrestbot wb2-phab, wb2-irc

=== May 19 ===
* 18:50 ircnotifier: valhallasw: Deployed f9b9d5bda60b9f1f6aac196254f8b6cfff6d58a2 Send Graph-VE MW extension project to VE channel wb2-phab, wb2-irc

=== May 2 ===
* 12:37 wikibugs: Updated channels.yaml to: f9b9d5bda60b9f1f6aac196254f8b6cfff6d58a2 Send Graph-VE MW extension project to VE channel

=== May 1 ===
* 22:10 wikibugs: Updated channels.yaml to: 831099cc50dbc6828c2ef5ff8f2e6aa41cd97310 Put a few things into -editing.
* 21:55 wikibugs: Updated channels.yaml to: b6c7fa03a61f5b27061be11900b6e432d500b765 Remove definitions for #wikimedia-mobile

=== April 29 ===
* 16:43 ircnotifier: valhallasw: Deployed 1d785dc3ad22a434749f8ec0d466180f3de9ea52 channels: Continuous-Integration is now Continuous-Integration-Infrastructure wb2-phab, wb2-irc

=== April 24 ===
* 21:14 wikibugs: Updated channels.yaml to: 1d785dc3ad22a434749f8ec0d466180f3de9ea52 channels: Continuous-Integration is now Continuous-Integration-Infrastructure

=== April 21 ===
* 03:39 ircnotifier: legoktm: Deployed 8e88fc89deaa41b2a720845f5d20aa871ffa09d9 Add Blueprint skin to notify list for #wikimedia-design wb2-irc
* 03:39 ircnotifier: legoktm: Deployed 8e88fc89deaa41b2a720845f5d20aa871ffa09d9 Add Blueprint skin to notify list for #wikimedia-design wb2-phab

=== April 20 ===
* 11:18 wikibugs: Updated channels.yaml to: 8e88fc89deaa41b2a720845f5d20aa871ffa09d9 Add Blueprint skin to notify list for #wikimedia-design

=== April 18 ===
* 20:24 valhallasw`cloud: file system corruption?? channels.yaml is all \x00s and .git/objects/* is corrupt. Cleared .git/objects, git fetch --all'd and git checkout channels.yaml seems to bring wikibugs back to life
* 19:43 valhallasw`cloud: tools-redis doesn't respond to commands, which could explain why wb2-phab was hanging. But why is tools-redis completely broken?
* 19:41 valhallasw`cloud: now wb2-phab is functioning again, but wb2-irc is not reporting?! Restarting that as well
* 19:38 valhallasw`cloud: that is, the last message to irc. The bot is still running and doing ping/pongs. However, wikibugs.log is completely silent after that time. wb2-phab.err does have errors, but without timestamps, so it's basically useless. Restarting wb2-phab to see if that helps
* 19:36 valhallasw`cloud: last message in redis2irc.log was 2015-04-18 02:10:26,157
* 19:36 valhallasw`cloud: wikibugs has broken down again. Trying to figure out why.

=== April 13 ===
* 19:05 wikibugs: Updated channels.yaml to: 4500101f021b8eec83899848932edaee98bd680a Merge &quot;Tools-Labs-xTools to #xtools&quot;

=== April 7 ===
* 16:13 wikibugs: Updated channels.yaml to: 8a4346f6c5d0f826a9b3099d5f76339d7a64dcad Merge &quot;Remove Quality Assurance from -releng&quot;

=== April 1 ===
* 20:56 wikibugs: Updated channels.yaml to: f09815aee08458b7fb283db7c7e0aed49e3b149d HACK: Always join channels on privmsg
* 20:56 ircnotifier: legoktm: Deployed f09815aee08458b7fb283db7c7e0aed49e3b149d HACK: Always join channels on privmsg wb2-irc

=== March 30 ===
* 19:45 wikibugs: Updated channels.yaml to: b2c38567b32d82881baab5c3227f14a9b8e9fff5 Send MediaWiki-API-Team and Blocked-on-MediaWiki-API-Team to #mediawiki-core

=== March 23 ===
* 18:44 wikibugs: Updated channels.yaml to: 90eed2a902164a9a1cf7930c7d9fb599ec9ae660 Send Commons to #wikimedia-commons-tech, per Steinsplitter

=== March 18 ===
* 11:49 ircnotifier: yuvipanda: Deployed 23240bd0dc5aebcc2a94b6f1ac268e2e3ad41114 Add more projects for devtools and mobile wb2-phab, wb2-irc

=== March 16 ===
* 17:16 wikibugs: Updated channels.yaml to: 23240bd0dc5aebcc2a94b6f1ac268e2e3ad41114 Add more projects for devtools and mobile

=== March 13 ===
* 22:29 legoktm: restarted wb2-irc to see if it rejoins channels properly
* 20:44 wikibugs: Updated channels.yaml to: 9eafe437ff005a3232e7f7e89dbb2be54437f76c tox: Rename channels env to standard py34

=== March 11 ===
* 04:36 legoktm: restarted both wb2-phab and wb2-irc

=== March 10 ===
* 19:30 wikibugs: Updated channels.yaml to: 614ee42338f6ab3f8d0705d3f0358523189af00e send WMT bugs to #wmt

=== March 9 ===
* 21:11 ircnotifier: legoktm: Deployed 614ee42338f6ab3f8d0705d3f0358523189af00e send WMT bugs to #wmt wb2-irc
* 20:36 wikibugs: Updated channels.yaml to: 614ee42338f6ab3f8d0705d3f0358523189af00e send WMT bugs to #wmt
* 20:26 wikibugs: Updated channels.yaml to: 614ee42338f6ab3f8d0705d3f0358523189af00e send WMT bugs to #wmt
* 20:18 ircnotifier: legoktm: Deployed 614ee42338f6ab3f8d0705d3f0358523189af00e send WMT bugs to #wmt wb2-irc

=== March 8 ===
* 19:14 wikibugs: Updated channels.yaml to: 614ee42338f6ab3f8d0705d3f0358523189af00e send WMT bugs to #wmt

=== March 3 ===
* 20:02 wikibugs: Updated channels.yaml to: 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge &quot;Use brown instead of red for orange (=release) projects&quot;
* 16:44 ircnotifier: legoktm: Deployed 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge &quot;Use brown instead of red for orange (=release) projects&quot; wb2-irc

=== February 28 ===
* 22:57 valhallasw`cloud: also restart wikibugs; it seems the PRIVMSGs in the log don't actually show up on irc
* 22:53 valhallasw`cloud: false alarm, messages were reported (2015-02-28 22:50:43,202 - irc3.wikibugs - DEBUG - &gt; PRIVMSG #mediawiki-parsoid :10Parsoid, 10VisualEditor, 10VisualEditor-EditingTools, etc) which is a few minutes ago
* 22:52 valhallasw`cloud: restarted wb2-phab to see if we get stuff from phab again

=== February 24 ===
* 23:16 ircnotifier: legoktm: Deployed 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge &quot;Use brown instead of red for orange (=release) projects&quot; wb2-irc
* 23:16 ircnotifier: legoktm: Deployed 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge &quot;Use brown instead of red for orange (=release) projects&quot; wb2-phab

=== February 22 ===
* 22:54 ircnotifier: valhallasw: Deployed 6da78462504cd023e0c31babb5cc56a7eae3a88a Merge &quot;Use brown instead of red for orange (=release) projects&quot; wb2-irc
* 21:44 ircnotifier: legoktm: Deployed 1f579477957417a693308fa9a23d2080821eb551 Volunteer? --&gt; Lowest (priority) wb2-irc

=== February 20 ===
* 20:04 wikibugs: Updated channels.yaml to: 26b8b9f5f812b092b02d33b3b29cf448dafb663a More fundraising projects to #wikimedia-fundraising
* 18:46 ircnotifier: valhallasw: Deployed 4cf33271a75d3655addac95cc16413ab1adc6488 Merge &quot;Always show four tags, most relevant first&quot; wb2-irc

=== February 18 ===
* 17:31 ircnotifier: legoktm: Deployed 8ba77ed2d2c039a231f3265da01215e721480ce0 Merge &quot;Log ALL the things!&quot; wb2-phab

=== February 17 ===
* 22:32 wikibugs: Updated channels.yaml to: 8ed0a167e287b7c3374f8b9b7e556e9b4b6180d6 Send AutoWikiBrowser to #autowikibrowser

=== February 16 ===
* 13:51 valhallasw`cloud: reverted locally (git revert &lt;new formatting commit&gt;) and restarted as it was breaking people's workflows
* 04:35 wikibugs: Updated channels.yaml to: eb4a51a4628a6b26d4e798b3e55d8749231bb72c Add Blocked-on-RelEng to -releng
* 00:54 ircnotifier: legoktm: Deployed 4c82585a9c01bceeb91acabbc5b481ea5928327d Merge &quot;Send Wikibase stuff to #wikidata&quot; wb2-irc
* 00:53 ircnotifier: legoktm: Deployed 4c82585a9c01bceeb91acabbc5b481ea5928327d Merge &quot;Send Wikibase stuff to #wikidata&quot; wb2-phab
* 00:05 ircnotifier: legoktm: Deployed d5922a4d10169ec8870e55de1b74ea9e39dc8c5c Make sure URL is always present wb2-phab
* 00:05 ircnotifier: legoktm: Deployed d5922a4d10169ec8870e55de1b74ea9e39dc8c5c Make sure URL is always present wb2-irc

=== February 13 ===
* 18:26 ircnotifier: valhallasw: Deployed 0941e5af42ab1c035b023246da5dde30b17c0f63 Remove Phabricator and Code-Review from -releng wb2-irc
* 18:01 ircnotifier: legoktm: Deployed 0941e5af42ab1c035b023246da5dde30b17c0f63 Remove Phabricator and Code-Review from -releng wb2-phab
* 17:30 ircnotifier: legoktm: Deployed 0941e5af42ab1c035b023246da5dde30b17c0f63 Remove Phabricator and Code-Review from -releng wb2-irc

=== February 11 ===
* 17:53 wikibugs: Updated channels.yaml to: 0941e5af42ab1c035b023246da5dde30b17c0f63 Remove Phabricator and Code-Review from -releng

=== February 7 ===
* 22:38 ircnotifier: valhallasw: Deployed 6d78d47f1eae25b63f4cd322a6737db58b8d5c7a Rework logging infrastructure wb2-phab, wb2-irc

=== February 6 ===
* 19:42 ircnotifier: legoktm: Deployed 1b6bbd391ad1f23a8270d3547b2540064e452d94 Fix project tag screen scraping wb2-phab

=== February 5 ===
* 22:51 ircnotifier: valhallasw: Deployed d9a83a0d71b0dd4500d40ebba5232b2ded362be5 Assume channel list is utf-8 wb2-irc
* 22:13 ircnotifier: valhallasw: Deployed 9054845f4a69a7364f5270e2ada574f696e4f70f Add MoodBar to wikimedia-collaboration wb2-phab

=== February 3 ===
* 06:05 ircnotifier: legoktm: Deployed 9054845f4a69a7364f5270e2ada574f696e4f70f Add MoodBar to wikimedia-collaboration wb2-phab
* 05:39 wikibugs: Updated channels.yaml to: 9054845f4a69a7364f5270e2ada574f696e4f70f Add MoodBar to wikimedia-collaboration

=== February 2 ===
* 19:08 wikibugs: Updated channels.yaml to: 490e8ba1784e8ef7b04d2f51d2697f1d670d6cb1 Announce Staging bugs to -releng

=== January 30 ===
* 04:28 wikibugs: Updated channels.yaml to: 4fe2e5b9f9d699d3547aba5b320fdf9ce1bd96b0 Send fundraising stuff to our channel

=== January 28 ===
* 14:29 wikibugs: Updated channels.yaml to: 9f4845ee4937cc9bf890bc7ea2251ca6613080e0 Merge &quot;Labs-Team was renamed to Labs&quot;

=== January 22 ===
* 22:21 wikibugs: Updated channels.yaml to: 6e130fecc19a39a5caed12ca0dda25ad28df62f0 -WikidataRepo is now -WikidataRepository

=== January 19 ===
* 20:52 valhallasw: is this really broken? :(

=== January 15 ===
* 20:03 ircnotifier: legoktm: Deployed c61edcfab64d62081edc3ccf89534764017f4a1c Make sure we're in the channel before messaging it wb2-irc

=== January 14 ===
* 22:52 ircnotifier: legoktm: Deployed 492438a4da3bd10a6e53bd248c997a02edb9d781 Fix wikibugs after Phabricator update wb2-phab
* 12:17 ircnotifier: yuvipanda: Deployed 9521ec19491d35ebc40fdccb34e75e0bd7f9399f Turn ssl off wb2-phab, wb2-irc
* 12:10 ircnotifier: yuvipanda: Deployed 8736032750b4fead35646ea9120621bf9d0ccb7e Only join actual channels wb2-phab, wb2-irc
* 12:09 ircnotifier: yuvipanda: Deployed 8736032750b4fead35646ea9120621bf9d0ccb7e Only join actual channels wb2-phab, wb2-irc
* 11:05 ircnotifier: yuvipanda: Deployed 2b66af26ca2a7343d0743423a4c9fcc6b8296e5e Disentangle tag lists for filtering vs display wb2-phab, wb2-irc
* 11:02 ircnotifier: yuvipanda: Deployed 2b66af26ca2a7343d0743423a4c9fcc6b8296e5e Disentangle tag lists for filtering vs display wb2-phab, wb2-irc

=== January 12 ===
* 22:04 wikibugs: Updated channels.yaml to: f1ee8fb8bc64186a1613ec9f9faf0aef6315759a Merge &quot;team-practices -&gt; #wikimedia-teampractices&quot;

=== January 11 ===
* 00:06 wikibugs: Updated channels.yaml to: 2257da8655036ce4555e88c01dad4a85f0b7946e WM-Bot -&gt; #wm-bot

=== January 10 ===
* 22:26 wikibugs: Updated channels.yaml to: f6b5ed8212e566b726a59de69a707ebea6c70d4e Remove -qa from announce list (moved to -releng)
* 18:06 wikibugs: Updated channels.yaml to: 09630c2cd5beead10c0ab2bfd8df84e7337a0208 LabsDB-Auditor -&gt; labs

=== January 9 ===
* 18:28 wikibugs: Updated channels.yaml to: 4c4fc344a850a36ef47a7c9965c853a27863baac Make sure to reset to origin/master, and show current sha1 before doing so

=== January 8 ===
* 10:59 wikibugs: Updated channels.yaml to: 29b1c027a31c7650094b195e70e3a4ac82c05d00 Merge &quot;Add Wikibugs to -labs&quot;
* 09:51 wikibugs: Updated channels.yaml to: 019f6b0366a97df69733f7c80303aec8058ecb79 Wikibugs should listen to the Multimedia project for the multimedia channel

=== January 7 ===
* 23:32 wikibugs: Updated channels.yaml to: 9538cc69ef4226d248a38fa86dadca6d646b6b37 Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2
* 15:29 wikibugs: Updated channels.yaml to: cc8bc876e23c6b58f06a7379273f34e858b6ade5 Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2

=== January 5 ===
* 21:32 wikibugs: Updated channels.yaml to: 9003536427ce097a62c3a8cec310f7ca4f0edab0 Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2
* 20:52 wikibugs: Updated channels.yaml to: c45fa33e94f5a34fa7618eaad1669104ace2a342 Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2

=== December 31 ===
* 16:48 wikibugs: Updated channels.yaml to: 0ba0b2c47cd593b64c4149931bfdaf022dff230c Merge branch 'master' of https://github.com/wikimedia/labs-tools-wikibugs2

=== December 22 ===
* 23:18 ircnotifier: legoktm: Deployed 101deca5ee3884d19a61fe7098f8296ddb0c43e0 Escape newlines in IRC output wb2-irc

=== December 18 ===
* 20:47 wikibugs: Updated channels.yaml to: 9ad8e090c8fb06d487b89255562483f08cf354e3 Send Spam-* to /dev/null
* 20:27 ircnotifier: legoktm: Deployed 3dc8fd7f3f8fdaacec5998913278179382b8594f Report IRC using Python and Yuvi's ircnotifier wb2-irc

=== December 17 ===
* 00:39 wm-bot: legoktm: Deployed 8502072659ddb8c55ae45026d54867771e3122e7 redis2irc: join channels after reloading config wb2-irc
* 00:28 wm-bot: legoktm: Deployed 8502072659ddb8c55ae45026d54867771e3122e7 redis2irc: join channels after reloading config wb2-irc

=== December 16 ===
* 23:35 wikibugs: Updated channels.yaml to: 432e66a45273e9798e26a8df08caa5a102eeec97 Add #wikimedia-services reporting
* 22:05 wm-bot: valhallasw: Deployed 3ec300c6605ed2087ad6bf25bf43abb4c0319d18 fab: set use_ssh_config = True (no jobs restarted)
* 21:46 wm-bot: valhallasw: Deployed 366f1b524cb4aecbdf4825a8b96e9f66524fa727 Add fabric runner wb2-phab
* 21:14 wikibugs: Updated channels.yaml to: 9649aa14cf1b8fd63a0e6efd3ac1aff0c351b141 Auto-detect changes to channels.yaml and !log it
* 21:06 wm-bot: valhallasw: Deployed 9649aa14cf1b8fd63a0e6efd3ac1aff0c351b141 wb2-phab, wb2-irc
* 20:44 legoktm: restarting for https://gerrit.wikimedia.org/r/180245

=== December 10 ===
* 19:18 legoktm: restarting phab listener for https://gerrit.wikimedia.org/r/178880
* 19:16 legoktm: restarting to pick up https://gerrit.wikimedia.org/r/178874 https://gerrit.wikimedia.org/r/178880

=== December 9 ===
* 19:46 legoktm: restarting for https://gerrit.wikimedia.org/r/178578
* 19:22 legoktm: restarting for https://gerrit.wikimedia.org/r/178561 https://gerrit.wikimedia.org/r/178563

=== December 4 ===
* 02:48 legoktm: restarted for https://gerrit.wikimedia.org/r/177372

=== December 1 ===
* 14:31 YuviPanda: killed irc bot for now

=== November 29 ===
* 21:25 legoktm: restarting for https://gerrit.wikimedia.org/r/176486
* 21:14 legoktm: restarting for https://gerrit.wikimedia.org/r/176483 
* 14:19 valhallasw`cloud: deployed 0a6dedd75e203f5005a15e340b2fed5ba4c67224

=== November 25 ===
* 23:21 legoktm: restarted wikibugs.py listener for https://gerrit.wikimedia.org/r/175890
* 00:07 legoktm: restarting wikibugs for -qa changes

=== November 24 ===
* 18:29 legoktm: restarting wikibugs for https://gerrit.wikimedia.org/r/175474
* 02:54 legoktm: RIP pywikibugs

=== November 18 ===
* 11:07 valhallasw`cloud: deployed https://github.com/legoktm/wikibugs2/commit/842d2d25a827dd2311ed98d1e4cd8af078bf10bb

=== October 11 ===
* 20:14 legoktm: deploying https://gerrit.wikimedia.org/r/166217 

=== September 24 ===
* 04:45 legoktm: deployed https://gerrit.wikimedia.org/r/162201 (add OpenStackManager component to #wikimedia-labs)

=== August 19 ===
* 19:06 valhallasw`cloud: new version deployed (gerrit 143239 and 143238)

=== July 1 ===
* 16:47 valhallasw: restarted wikibugs with new channel config / https://gerrit.wikimedia.org/r/#/c/142992/ / Nemo_bis 

=== May 22 ===
* 19:14 valhallasw: changed git repo to have gerrit as master
* 12:27 valhallasw: wikibugs-l@l.wm.o delivery functional again and wikibugs is correctly reporting to IRC
* 12:18 valhallasw: gmail-to-wikibugs delivery is now functional; hopefully wikibugs-l@lists.wm.o delivery too...
* 12:17 valhallasw: mail delivery broken; direct mails complain about open(&quot;~/mailout.log&quot;, &quot;a&quot;) in to_redis.py; commented out those lines

=== April 30 ===
* 20:59 valhallasw: Deployed {{Gerrit|a48a000}}

=== April 28 ===
* 20:11 YuviPanda: restarted wikibugs, seems to have died
* 11:54 YuviPanda: deployed {{Gerrit|bf1be7b55a19457469f311ae54e1cf6409eb4a0b}}
* 06:42 valhallasw: deployed 2.0-1-gb7f4290
* 06:23 valhallasw: Merging and deploying {{Gerrit|b7bbf92d7d2ceef993afc1113f515e01d79e1248}}
* 06:21 valhallasw: NameError: name '_wsp_splitter' is not defined in /data/project/wikibugs/src/pywikibugs/get_unstructured.py. Apparently the line 'from email._header_value_parser import _wsp_splitter, _validate_xtext' had not made it into the git repo, and was cleared by accident on deploymeny
* 06:17 valhallasw: wikibugs stopped reporting; investigating

=== April 27 ===
* 18:39 valhallasw: Updated wikibugs to {{Gerrit|245f2a2}}
* 18:36 valhallasw: Updated wikibugs to {{Gerrit|407ad66}}


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>676ggm14knc4ptiygv4h588ufoqm6b6</sha1>
    </revision>
  </page>
  <page>
    <title>User:JM28~labswiki/Books/How to</title>
    <ns>2</ns>
    <id>23285</id>
    <revision>
      <id>2414316</id>
      <parentid>2275513</parentid>
      <timestamp>2026-05-16T19:57:23Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <origin>2414316</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="18" sha1="eyy6jdot5p775art85i3kskiyp0d713" xml:space="preserve">:[[Server Spares]]</text>
      <sha1>eyy6jdot5p775art85i3kskiyp0d713</sha1>
    </revision>
  </page>
  <page>
    <title>Prometheus</title>
    <ns>0</ns>
    <id>202567</id>
    <revision>
      <id>2414328</id>
      <parentid>2385896</parentid>
      <timestamp>2026-05-16T22:17:52Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414328</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="39095" sha1="barauyq4uttaaprbmf0hdzsqgw4iusa" xml:space="preserve">{{Navigation Wikimedia infrastructure|expand=logging}}
'''Prometheus''' is an open source software ecosystem for monitoring and alerting, with focus on reliability and simplicity. See also upstream's [https://prometheus.io/docs/introduction/overview/ Prometheus overview] and [https://prometheus.io/docs/introduction/faq/ Prometheus FAQ].

== What is it? ==
Distinguishing features of Prometheus compared to other metrics systems include:

; multi-dimensional data model
: Metrics have a &lt;tt&gt;name&lt;/tt&gt; and several &lt;tt&gt;key=value&lt;/tt&gt; pairs to better model what the metric is about. e.g. to measure varnish requests in the upload cache in eqiad we'd have a metric like &lt;tt&gt;http_requests_total{cache=&quot;upload&quot;,site=&quot;eqiad&quot;}&lt;/tt&gt;.

; a [https://prometheus.io/docs/prometheus/latest/querying/basics/ powerful query language] (PromQL)
: Makes it able to ask complex questions, e.g. when debugging problems or drilling down for root cause during outages. From the example above, the query &lt;tt&gt;topk(3, sum(http_requests_total{status~=&quot;^5&quot;}) by (cache))&lt;/tt&gt; would return the top 3 caches (text/upload/misc) with the most errors (status matches the regexp &quot;^5&quot;)

; pull metrics from targets
: Prometheus is primarily based on a ''pull'' model, in which the prometheus server has a list of ''targets'' it should ''scrape'' metrics from. The pull protocol is HTTP based and simply put, the target returns a list of &quot;&lt;metric&gt; &lt;value&gt;&quot;. Pushing metrics is supported too, see also http://prometheus.io/docs/instrumenting/pushing/.

== Service ==

=== Server location ===
The various Prometheus servers are logically separated, though physically they can share one or multiple hosts. As of April 2022, we run Prometheus on baremetal hardware in [[Eqiad data center|Eqiad]] and [[Codfw data center|Codfw]], and on [[Ganeti]] VMs in all POPs / caching centers. The list of prometheus instances can be found in [[gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/prometheus.yaml|hieradata/common/prometheus.yaml]] and the biggest instances (ops, k8s-related) are spread amongst multiple hosts in eqiad and codfw.

''Note: there are additional Prometheus instances running in [[Cloud VPS]] and managed by the [[Wikimedia Cloud Services team]], these are documented in [[Help:Cloud VPS managed monitoring]].''

== Architecture ==
Each Prometheus server is configured to ''scrape'' a list of ''targets'' (i.e. HTTP endpoints) at a certain frequency, in our case defaulting to 60s. All metrics are stored on the local disk with a per-instance multi-week retention period.

All targets to be scraped are grouped into ''jobs'', depending on the purpose that those targets serve. For example the job to scrape all host-level data for a given location using &lt;tt&gt;node-exporter&lt;/tt&gt; will be called &lt;tt&gt;node&lt;/tt&gt; and each target will be listed as &lt;tt&gt;hostname:9100&lt;/tt&gt;. Similarly there are jobs for varnish, mysql, etc.

Each Prometheus server is meant to be stand-alone and polling targets in the same failure domain as the server itself as appropriate (e.g. the same site, the same vlan and so on). For example this allows to keep the monitoring local to the site and not have spotty metrics upon cross-site connectivity blips. 

[[File:Prometheus_single_server.png]]

=== Exporters ===
The endpoint being polled by the prometheus server and answering the &lt;code&gt;GET&lt;/code&gt; requests is typically called ''exporter'', e.g. the host-level metrics exporter is ''node-exporter''.

Each exporter serves the current snapshot of metrics when polled by the prometheus server, there is no metric history kept by the exporter itself. Further, the exporter usually runs on the same host as the service or host it is monitoring.

=== Pushgateway ===
For ephemeral jobs that are not long-lived enough to be scraped via an HTTP endpoint there's a push mechanism available called the [https://github.com/prometheus/pushgateway/blob/master/README.md Prometheus pushgateway] 

Prometheus Pushgateway (sometimes written as push-gateway) allows metrics to be pushed via HTTP to a gateway that is subsequently scraped by Prometheus once per minute (as opposed to the traditional exporter scrape/pull model described above) 
{| class=&quot;wikitable&quot;
|+
|'''WMF hosts a production Prometheus pushgateway at''' &lt;tt&gt;http://prometheus-pushgateway.discovery.wmnet&lt;/tt&gt;&lt;small&gt;&lt;br /&gt;''Use pushgateway with care -- '''Please read [https://prometheus.io/docs/practices/pushing/ best practices on when to use the pushgateway] before deploying a new service.'''''&lt;/small&gt;
|}{{Warning|1=When using TLS for metric scraping, make sure the host on the certificate and the one configured match, or you will get a TLS Handshake error. By default, puppet sets just the hostname as the target of monitoring -you are likelty to want to add the option &lt;code&gt;hosts_only =&gt; false&lt;/code&gt; to use the full qualified domain name as target}}

=== Storage ===
Why just stand-alone prometheus servers with local storage and not clustered storage? The idea behind a single prometheus server is one of reliability: a monitoring system must be ''more'' reliabile than the systems it is monitoring. It is certainly easier to get local storage right and reliable than clustered storage, especially important when collecting operational metrics.

See also [https://prometheus.io/docs/operating/storage/ prometheus storage documentation] for a more in-depth explanation and storage space requirements.

=== High availability ===
With local storage being the basic building block we can still achieve high-availability by running more than one server in parallel, each configured the same and polling the same set of targets. Queries for data can be routed via LVS in an active/standby fashion, under normal circumstances the load is shared (i.e. active/active).

[[File:Prometheus_HA_server.png]]

=== Recording Rules &amp; Alerting Rules ===
&quot;''Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. Querying the precomputed result will then often be much faster than executing the original expression every time it is needed. This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh.''

''Recording and alerting rules exist in a rule group. Rules within a group are run sequentially at a regular interval, with the same evaluation time. The names of recording rules must be valid metric names. The names of alerting rules must be valid label values.''&quot; [https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/].

For naming conventions, we follow the Prometheus best practices: [https://prometheus.io/docs/practices/rules/].

In our deployment we maintain recording and alerting rules in two locations:

'''Prometheus rules''' are deployed for a specific prometheus instance, these are limited in scope to that prometheus instance e.g. ops, k8s, etc. and '''generally more performant than Thanos rule'''.  These recording rules are managed with &lt;code&gt;prometheus::rule&lt;/code&gt; in puppet, and alerting rules via the [[gerrit:q/project:operations/alerts|operations/alerts]] gerrit.
Currently, it is possible to find some recording rule implementations at:
* [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/prometheus/rules_analytics.yml analytics]
* [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/prometheus/rules_services.yml services]
* [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/prometheus/rules_ops.yml ops]
* [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/prometheus/rules_k8s.yml k8s]
* [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/prometheus/rules_ext.yml ext]

'''Thanos rule recording rules''' are recording rules executed by [https://thanos.io/tip/components/rule.md/ Thanos rule (aka ruler)].  These rules query thanos-query and as such are able to cross prometheus instance boundaries.  Note: at the present time we have two ruler instances (main and pilot) and may deploy more instances in the future.  Thanos recording rules are managed with &lt;code&gt;thanos::recording_rule&lt;/code&gt; in puppet, and in addition to that our SLO tooling dynamically writes recording rules directly to the rule instance. Recording rule definitions for the Thanos Ruler component can be found in [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/files/thanos/recording_rules.yaml recording_rules.yaml].

==== How to Add a Recording Rule (with Example) ====
The ''mediawiki_CentralAuth_session_read_seconds_bucket'' metric has very high cardinality [211,974 series (i.e: distinct label sets) on 2/3/2026].

Computing range-vector functions (''rate'', ''increase'') and/or aggregations (''sum'', ''avg'') and/or ''&lt;aggregation&gt;_over_time'' could be computationally infeasible on such metrics.

On a Grafana dashboard a widget is plotting:
&lt;pre&gt;
histogram_quantile(0.50, sum by(le) (rate(mediawiki_CentralAuth_session_read_seconds_bucket[5m])))
&lt;/pre&gt;

Considering how much data would be retrieved and aggregated over a window of weeks (or months) for this metric—which could have 200k series for an instant query—gives an idea of the computational effort involved.

So, given that this metric is coming from the k8s Prometheus instance, it is possible to add a recording rule to precompute the needed value:
&lt;pre&gt;
# modules/profile/files/prometheus/rules_k8s.yml
...
- name: mediawiki_centralauth
  rules:
    - record: le:mediawiki_CentralAuth_session_read_seconds_bucket:rate5m
      expr: sum by (le) (rate(mediawiki_CentralAuth_session_read_seconds_bucket[5m]))
...
&lt;/pre&gt;

According to [https://prometheus.io/docs/practices/rules/ Prometheus best practices] regarding recording rules, the name of the output metric (contained in the ''record:'' field) will follow the schema ''level:metric:operations'':
* ''level'' represents the aggregation level and labels of the rule output: ''le'';
* ''metric'' is the metric name and should be unchanged other than stripping _total off counters when using rate() or irate(): ''mediawiki_CentralAuth_session_read_seconds_bucket'';
* ''operations'' is a list of operations that were applied to the metric, newest operation first: ''rate5m''.

Once the patch is ready to be submitted, please add team members from [https://office.wikimedia.org/wiki/Team_interfaces/SRE_-_Observability o11y] as reviewers.

The expr on Grafana widget could now been rewritten as:
&lt;pre&gt;
histogram_quantile(
    0.50
  ,
    sum by(le) (le:mediawiki_CentralAuth_session_read_seconds_bucket:rate5m)
)
&lt;/pre&gt;

The prepending ''sum by (le)'' is still needed since Grafana retrieves data from Thanos (and therefore from all the ''k8s'' Prometheus instances). A further aggregation of all data from all instances is required to achieve the same result as before.

To avoid dealing with the aggregation of recording rules coming from different Prometheus instances across the Foundation, it is possible to add the recording rule on the Thanos side, keeping in mind that deploying directly on Prometheus instances is typically more performant in terms of resource consumption than on Thanos.

=== Backups ===
For efficiency reasons, prometheus spools chunks of datapoints in memory for
each metric before flushing them to disk. This makes it harder to perform
backups online by simply copying the files on disk. The issue of having
consistent backups is also discussed in
[https://github.com/prometheus/prometheus/issues/651 prometheus #651].

Notwithstanding the above, it should be possible to backup the prometheus local
storage files as-is by archiving its storage directory with tar before regular
(bacula) backups. Since the backup is being done online it will result in some
inconsistencies, upon restoring the backup Prometheus will crash-recovery its storage at startup.

To perform backups of consistent/clean state, at the moment prometheus needs to
be shutdown gracefully, therefore when running an active/standby configuration backup can be taken on the standby prometheus to minimize its impact. Note that
the shutdown will result in gaps in the standby prometheus server for the duration of the shutdown.

=== Failure recovery ===
In the event of a prometheus server having an unusable local storage (disk
failed, filesystem failed, corruption, etc) failure recovery can take the form of:

* start with empty storage: of course it is a complete loss of metric history for the local server and will obviously fully recover once the metric retention period has passed.

* recover from backups: restore the storage directory to the last good backup

* copy data from a similar server: when deployed in pairs it is possible to copy/rsync the storage directory onto the failed server, this will likely result in gaps in the recent history though (see also Backups)

=== Meta-Monitoring ===
The Thanos/Prometheus stack is now meta-monitored using continuous testing techniques. The results of these tests are collected by a centralized script running on the Alertmanager hosts and exposed via an HTTP endpoint for external monitoring tools.

Related tasks:
* [https://phabricator.wikimedia.org/T281812 Audit/Assess meta monitoring strategy]
* [https://phabricator.wikimedia.org/T397003 Configure a prometheus dead man's snitch alert]

==== Dead Man Switch ====
On every Prometheus (and Thanos) instance, an [https://training.promlabs.com/training/monitoring-and-debugging-prometheus/metrics-based-meta-monitoring/end-to-end-watchdog-alerts/ always-firing alert] (DeadManSwitch) is deployed. The alert deployment is managed by Puppet, so it is tightly coupled with the instance configuration.

Each Prometheus (and Thanos) instance evaluates its own rules, and the alerts are sent to Alertmanager. Then, Alertmanager routes these alerts to a dedicated hook, which simply touches a file on the file system.

==== Public Endpoints ====
The public endpoints are implemented through a Python/Flask application, exposed via Apache and uWSGI on both the active and passive Alertmanager hosts. These endpoints are organized by service (Thanos/Prometheus) and check type (module)(deadmanswitchnotified/deadmanswitchonamdb).
There are three hostnames that publish the service:
* metamonitoring.wikimedia.org (always associated with the active instance)
* metamonitoring-active.wikimedia.org (always associated with the active instance)
* metamonitoring-passive.wikimedia.org (always associated with the passive instance)

The final URL takes the following form:
&lt;pre&gt;
https:\/\/metamonitoring(-(active|passive))?\.wikimedia\.org\/(prometheus|thanos)\/(deadmanswitchonamdb|deadmanswitchnotified)
&lt;/pre&gt;

===== deadmanswitchnotified module [Alertmanager: (Prometheus|Thanos) Watchdog Hook Processing check] =====
This module is used to verify whether any DeadManSwitch alert processed by the Alertmanager hook is recent, within a specified time frame.
It is intended to be called only on metamonitoring.wikimedia.org.

''Alerts description on HetrixTools:''
* Alertmanager: Prometheus Watchdog Hook Processing check
* Alertmanager: Thanos Watchdog Hook Processing check

===== deadmanswitchonamdb module [Alertmanager-(active|passive):(Prometheus|Thanos) Watchdog Alerts Presence] =====
This module is used to verify whether all DeadManSwitch alerts are present in the Alertmanager database. Since its purpose is also to ensure the correct alignment of the database between active and passive hosts, it is intended to be called on both of the following endpoints:
* metamonitoring-active.wikimedia.org;
* metamonitoring-passive.wikimedia.org.

''Alerts description on HetrixTools:''
* Alertmanager-active: Prometheus Watchdog Alerts Presence check
* Alertmanager-active: Thanos Watchdog Alerts Presence check
* Alertmanager-passive: Prometheus Watchdog Alerts Presence check
* Alertmanager-passive: Thanos Watchdog Alerts Presence check

===== HetrixTools =====
HetrixTools is an external monitoring service configured to query the previously defined endpoints from various global locations. In case of an incident, alerts are routed to VictorOps. Access credentials for the tool can be retrieved via pwstore.

===== Troubleshooting =====
To get more details about the alerts, you can either manually query the endpoint—since the response body includes the list of instances that aren’t working as expected—or check the daemon log.
&lt;pre&gt;
root@alert1002:~# journalctl -u uwsgi-metamonitoring_public_endpoint.service
...
Aug 14 22:03:21 alert1002 gunicorn[864530]: INFO - deadmanswitchnotified - /var/lib/o11y-metamonitoring/deadmanswitchamhook/prometheus_ops_eqsin has a timestamp older than 600
...
&lt;/pre&gt;

== Service Discovery ==
Prometheus supports different kinds of discovery through its [https://prometheus.io/docs/operating/configuration/ configuration]. For example, in &lt;tt&gt;role::prometheus::labs_project&lt;/tt&gt; implements auto-discovery of all instances for a given labs project. 
&lt;code&gt;file_sd_config&lt;/code&gt; is used to continuously monitor a set of configuration files for changes and the script &lt;code&gt;prometheus-labs-targets&lt;/code&gt; is run periodically to write the list of instances to the relative configuration file. The &lt;code&gt;file_sd&lt;/code&gt; files are reloaded automatically by prometheus, so new instances will be auto-discovered and have their instance-level metrics collected.

While file-based service discovery works, Prometheus also supports higher-level discovery for example for Kubernetes (see also &lt;tt&gt;profile::prometheus::k8s&lt;/tt&gt;).

=== Relabel rules ===
Relabel rules (section &lt;tt&gt;relabel_config&lt;/tt&gt;, &lt;tt&gt;metric_relabel_configs&lt;/tt&gt;, etc) in service discovery can be a bit hard to follow. https://relabeler.promlabs.com/ can help in visualizing those.

== Adding new metrics ==
In general Prometheus' model is pull-based. In practical terms that means that once metrics are available over HTTP somewhere on the network with the methods described below, Prometheus itself should be instructed to poll for metrics via its configuration (more specifically, a ''job'' as described in [https://prometheus.io/docs/concepts/jobs_instances/ upstream documentation]). Within WMF's Puppet the Prometheus configuration lives inside its respective instance profile, for example &lt;code&gt;modules/profile/manifests/prometheus/ops.pp&lt;/code&gt; is often the right place to add new jobs.

=== Direct service instrumentation ===
The most benefits from service metrics are obtained when services are directly instrumented with one of Prometheus clients, e.g. [https://github.com/prometheus/client_python Python client].
Metrics are then exposed via HTTP/HTTPS, commonly at &lt;tt&gt;/metrics&lt;/tt&gt;, on the service's HTTP(S) port (in the common case) or a separate port if the service doesn't talk HTTP to begin with.

=== Service exporters ===
For cases where services can't be directly instrumented (aka whitebox monitoring), a sidekick application &lt;tt&gt;exporter&lt;/tt&gt; can be run alongside the service that will query the service using whatever mechanism and expose prometheus metrics via the client. This is the case for example for [https://github.com/jonnenauha/prometheus_varnish_exporter varnish_exporter] parsing &lt;tt&gt;varnishstat -j&lt;/tt&gt; or [https://github.com/neezgee/apache_exporter apache_exporter] parsing apache's &lt;tt&gt;mod_status&lt;/tt&gt; page.

=== Machine-level metrics ===
Another class of metrics is all those related to the machine itself rather than a particular service. Those involve calling a subprocess and parsing the result, often in a cronjob. In these cases the simplest thing to do is drop plaintext files on the machine's filesystem for &lt;tt&gt;node-exporter&lt;/tt&gt; to pick up and expose the metrics on HTTP. This mechanism is named &lt;tt&gt;textfile&lt;/tt&gt; and for example the python client has support for it, e.g. [https://github.com/prometheus/client_python#node-exporter-textfile-collector sample textfile collector usage]. This is most likely the mechanism we could use to replace most of the custom collectors we have for Diamond.

=== Network probes (blackbox exporter) ===
As of Jul 2022 it is possible to run so-called network blackbox probes via Prometheus. Said probes are run from Prometheus hosts themselves, target network services and are used to assert whether the service works from a user/client perspective (hence the &quot;blackbox&quot; terminology). Please note that blackbox probes will ''not'' reliably test for network access (e.g. firewall rules) since Prometheus hosts can access all hosts' ports internally.

If your service is part of &lt;tt&gt;service::catalog&lt;/tt&gt; in puppet then adding network probes is trivial in most cases. Add a &lt;tt&gt;probes&lt;/tt&gt; stanza to your service, for example probing &lt;tt&gt;/?spec&lt;/tt&gt; and test for a 2xx response is achieved by the following:

  probes:
    - type: http
      path: /?spec

Refer to the [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/wmflib/types/service/probe.pp Wmflib::Service::Probe type documentation] for more advanced use cases.

Custom checks/probes defined outside &lt;tt&gt;service::catalog&lt;/tt&gt; can be implemented in Puppet via &lt;tt&gt;prometheus::blackbox::check::{http,tcp,icmp}&lt;/tt&gt; abstractions. They will deploy both network probes and related alerts (e.g. when the probe is unsuccessful, or the TLS certificates are about to expire), by default probing both ipv4 and ipv6 address families. The probe's usage largely depends on the use case, ranging from a simple example like below:

  # Probe the phabricator.wikimedia.org vhost, using TLS, and talk to the host(s) this check is deployed to
  prometheus::blackbox::check::http { 'phabricator.wikimedia.org':
      severity =&gt; 'page',
  }

To more complex use cases like VTRS, checking responses for specific text, on ipv4, etc:

  prometheus::blackbox::check::http { 'ticket.wikimedia.org':
      team               =&gt; 'serviceops-collab',             
      severity           =&gt; 'warning',    
      path               =&gt; '/otrs/index.pl',
      port               =&gt; 1443,
      ip_families        =&gt; ['ip4'],      
      force_tls          =&gt; true,
      body_regex_matches =&gt; ['wikimedia'],
  }

Check [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/prometheus/manifests/blackbox/check/http.pp the http check documentation] for more information.

It is recommended to pick the highest-level check possible for your service (IOW prefer HTTP over TCP for example) to improve signal-to-noise ratio.

=== Metrics from Logs (prometheus-es-exporter) ===
Metrics can be generated from OpenSearch queries by defining a query in the [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/prometheus/files/es_exporter/ prometheus-es-exporter config].  To test, combine &lt;code&gt;00-default.cfg&lt;/code&gt; with your definition on a logstash &lt;code&gt;jobs_host&lt;/code&gt; and run
  $ prometheus-es-exporter --cluster-health-disable --nodes-stats-disable --indices-aliases-disable --indices-mappings-disable --indices-stats-disable --port 9999 --config-file my_file.cfg
and separately on the same host
  $ curl -s localhost:9999/metrics
If prometheus-es-exporter is successful parsing and running the new query, the curl output will contain your metrics in their final form.

== Example use ==
MySQL monitoring is performed by running &lt;code&gt;prometheus-mysqld-exporter&lt;/code&gt; on the database machine to be monitored. Metrics are exported via http on port &lt;code&gt;9104&lt;/code&gt; and fetched by prometheus server(s), to preview what metrics are being collected a fetch can be simulated with:&lt;pre&gt;curl -s localhost:9104/metrics | grep -v '^#'&lt;/pre&gt;Grafana dashboards:

* Per group / shard / role overview: https://grafana.wikimedia.org/d/000000278/mysql-aggregated
* Per server drilldown: https://grafana.wikimedia.org/d/000000273/mysql

== Query cheatsheet ==
'''Filter for a specific instance'''

Given values such as

 varnish_mgt_child_stop{instance=&quot;cp2001:9131&quot;,job=&quot;varnish-text&quot;,layer=&quot;backend&quot;}

and a template variable called '''$server''', containing the server hostname, one can filter for the selected instance as follows:

 varnish_mgt_child_start{instance=~&quot;$server:.*&quot;,layer=&quot;backend&quot;}

'''Filter by label using multi-values template variables'''

Given the following two metrics:

 varnish_version{job=&quot;varnish-upload&quot;, ...}
 node_uname_info{cluster=&quot;cache_upload&quot;, ...}

and a multi-value template variable called '''$cache_type''', with the following values: '''text,upload,misc,canary''', it is possible to write a prometheus query filtering the selected cache_types:
  
 varnish_version{job=~&quot;varnish-($cache_type)&quot;}
 node_uname_info{cluster=~&quot;cache_($cache_type)&quot;}

'''Dynamic, query-based template variables'''

Grafana's templating allows to define template variables based on Prometheus queries.

Given the following metric:
 node_uname_info{release=&quot;4.9.0-0.bpo.4-amd64&quot;, ...}
 node_uname_info{release=&quot;4.9.0-0.bpo.3-amd64&quot;, ...}

Choose '''Query''' as the variable Type, the desired '''Data Source''', and specify a query such as the following to extract the values:
  label_values(node_uname_info, release)

'''Query a metric with high accuracy even if with low precision (e.g. uptime)'''

Prometheus metrics will never provide high precision- this is mostly because scraping only happens every minute, resulting in values being accurate within that 1 minute of scrape time. However, there are times when you need high accuracy (getting the value at a specific time), even if you don't care what that time is. This is the case, for example, to calculate the uptime of a server: you don't care if you get stale results, as long as they are accurate in the past. To do so, you can query:

 timestamp(node_time_seconds) - node_time_seconds

This way the time will be accurate to the second. If you use the timestamp of the metric or &lt;code&gt;time()&lt;/code&gt;, you will get varying times within a minute.

== FAQ ==

=== How long are metrics stored in Prometheus? ===
As of June 2020, we have deployed [[Thanos]] for long term storage of metrics. The target retention period for all one-minute metrics is three years, although as of Jul 2022 the one-minute retention has been shortened for capacity reasons (cfr {{bug|T311690}}). Five-minute and one-hour aggregated datapoints retention target is still set at three years.

=== What are the semantics of rate/irate/increase? ===
These functions generally take a counter metric (i.e. non-decreasing) and return a &quot;value over time&quot;. Rate and irate return per second counts, while increase returns the change over the given interval. See also [https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates in depth explanation at promlabs.com]

=== What best practices should we use for label and metric naming ===
We generally tend to follow the same general guidelines as Prometheus: https://prometheus.io/docs/practices/naming/. Don't hesitate to reach out to Observability with further questions around metric/label naming.

=== How do I test/preview relabeling rules? ===
The relabeler online tool at https://relabeler.promlabs.com will help you understand what is going on

== {{Anchor|HOWTO}}Runbooks ==
=== Drop metrics by metric name ===
Occasionally, a metric will be created that exhibits high cardinality causing Thanos/Prometheus timeouts, full disks, and general instability.

To ban a metric by metric name, find the job the metric gets pulled in from and add a [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs metric_relabel_config] instructing Prometheus to drop the metric when encountered:

&lt;syntaxhighlight lang=&quot;puppet&quot;&gt;
'metric_relabel_configs' =&gt; [
  {
    'action'        =&gt; 'drop',
    'source_labels' =&gt; ['__name__'],
    'regex'         =&gt; 'metric_name.*'
  },
],
&lt;/syntaxhighlight&gt;

Note that the action will not apply if the action is added to &lt;code&gt;relabel_configs&lt;/code&gt;.

=== Global view (Thanos) web interface ===
As of Jul 2020 the [[Thanos]] web interface is available at https://thanos.wikimedia.org. This interface offers a global view over Prometheus data and should be preferred for new use cases. Please consult the [[Thanos]] page to find out more.

=== Access Prometheus web interface ===
Use https://thanos.wikimedia.org to run Prometheus queries across all Prometheus instances in all sites, metrics returned via Thanos have the prometheus and site extra labels, populated accordingly. Individual Prometheus web interface per-instance are available at https://prometheus-&amp;#x3C;site&amp;#x3E;.wikimedia.org/&amp;#x3C;instance&amp;#x3E;/ . For example to access 'ops' instance in 'eqiad': https://prometheus-eqiad.wikimedia.org/ops/

The Cloud Services team manage additional Prometheus deployments, hosted in Cloud VPS, for monitoring Cloud VPS and Toolforge: [[Portal:Cloud_VPS/Admin/Monitoring]].

=== List metrics with curl ===
One easy way to check what metrics are being collected by prometheus on a given machine is to request the metrics via HTTP like prometheus server does at scrape time, e.g. for node-exporter on port 9100:
  
  curl -s localhost:9100/metrics

=== Aggregate metrics from multiple sites ===
The use case for a &quot;global&quot; view of metrics used to be covered by the global Prometheus instance. Said instance is deprecated and this use case (and more) are covered by [[Thanos]].

=== Sync data from an existing Prometheus host ===
When replacing existing Prometheus hosts it is possible to keep existing data by rsync'ing the &lt;tt&gt;metrics&lt;/tt&gt; directory from the old host into the new. It is important to make sure first that the new host has puppet run successfully (thus Prometheus is configured) and can Prometheus can reach its targets successfully (i.e. the new host is part of &lt;tt&gt;prometheus_nodes&lt;/tt&gt; for its site. Once all of that is done the rsync can happen, on the new host:

  puppet agent --disable &quot;copying prometheus data&quot;
  export old_host=&lt;hostname&gt;
  export instance_name=ops
  systemctl stop prometheus@${instance_name}
  su -s /bin/bash prometheus
  prometheus-sync-data $old_host $instance_name

=== Prometheus host running out of space ===
It might happen that Prometheus hosts get close to running out of space on one of their per-instance filesystems. Assuming the underlying volume group has space available (&lt;tt&gt;lvs&lt;/tt&gt; to check what LVs are present and on which VGs, then &lt;tt&gt;vgs&lt;/tt&gt; to check VGs themselves) then it is possible to extend the filesystem online with (e.g. +25G to the prometheus-foo LV on vg0 VG, remove &lt;tt&gt;--test&lt;/tt&gt; once happy).

  lvextend --test --resizefs --size +25G vg0/prometheus-foo

Make sure to:
* Leave some space available on the VG, to handle cases like this in the future if possible
* Extend the filesystem on all prometheus hosts in the same site
* &lt;tt&gt;!log&lt;/tt&gt; your actions for easier traceability

=== No space available on the volume group ===

At some point the space on volume group might be fully allocated. In this case the emergency remedy is to decrease Prometheus retention time via &lt;tt&gt;prometheus::server::storage_retention&lt;/tt&gt; in Puppet, and restart Prometheus with the new settings.

In the unfortunate case that the filesystem is 100% utilized is also possible to manually remove storage &quot;blocks&quot; (i.e. directories) from the &lt;tt&gt;metrics&lt;/tt&gt; directory under &lt;tt&gt;/srv/prometheus/INSTANCE&lt;/tt&gt;. Sorting the filenames alphabetically will ensure they are sorted chronologically as well.

=== Add filesystems for a new instance ===
Until [[phab:T163692|T163692]] is fully resolved, new Prometheus instances require adding LVs to the Prometheus hosts in eqiad/codfw. When provisioning a new instance refer to &lt;tt&gt;modules/prometheus/files/provision-fs.sh&lt;/tt&gt;: add the new instance there and run the script on eqiad/codfw Prometheus hosts.

=== Add metrics from a new service ===
Most services which export metrics to Prometheus do so via an HTTP endpoint, running on its own port.  This HTTP endpoint can be served by the daemon itself, or by a separate &quot;exporter&quot; process.

Prometheus needs to be told to scrape the HTTP endpoint, which it calls a &quot;target.&quot;  (A logical grouping of targets is called a &quot;job.&quot;)  In addition to adding the new job to the Prometheus server, you will need to add a firewall rule exposing the HTTP endpoint.

For an example Puppet changes to add new jobs, see [[gerrit:c/operations/puppet/+/504360|change 504360]] and [[gerrit:#/c/operations/puppet/+/572141|change 572141]].

=== Stop queries on problematic instances ===
If a single Prometheus instance is misbehaving (e.g. overloaded) it is possible to temporarily stop queries from reaching that instance, by stopping Puppet commenting the relevant &lt;tt&gt;ProxyPass&lt;/tt&gt; entry in &lt;tt&gt;/etc/apache2/prometheus.d/&lt;/tt&gt; and issue &lt;tt&gt;apache2ctl graceful&lt;/tt&gt;. See also {{Bug|T217715}}.

[[Category:Bot and monitoring]]
[[Category:Runbooks]]

=== Prometheus was restarted ===
The alert on Prometheus uptime exists to notify opsen of the possibility of strange monitoring artifacts occurring, as [[Incident documentation/20190425-prometheus|has happened in the past]].  If it was just a single restart, and not a crashloop, no action is strictly necessary (but investigating what happened isn't a bad idea; Prometheus isn't supposed to crash or restart).

If this alert is firing for a 'global' Prometheus, it can mean that either the global instance restarted, or that one of the Prometheis scraped by the global instance restarted.

=== Configuration reload failure ===
Check for recent changes in Puppet, particular modifications to monitoring::check_prometheus invocations or to the underlying module/prometheus templates themselves. Hopefully the error message from Prometheus gives you some idea.

=== Prometheus job unavailable ===
As part of {{Bug|T187708}} there's alerting in place for unavailable Prometheus jobs. This alert means that Prometheus was unable to fetch metrics from most of the job's targets, usually for the following reasons:

* the targets themselves are effectively down, unreachable or fetching metrics timed out. Could be caused by missing firewall rules on the host, the service is down, etc
* the target files for Prometheus are incorrect or stale. For example Prometheus is trying to pull metrics from a port /service that's not provisioned on the host anymore. Check &lt;tt&gt;/srv/prometheus/INSTANCE/targets&lt;/tt&gt; on Prometheus hosts and the related Puppet configuration at &lt;tt&gt;modules/profile/manifests/prometheus/INSTANCE.pp&lt;/tt&gt;.

See also https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets for dashboard

=== Prometheus exporters &quot;up&quot; metrics unavailable ===
Some services don't have native Prometheus metrics support, thus an &quot;exporter&quot; is used that runs alongside the service and converts metrics from the service into Prometheus metrics. It might happen that the exporter itself is up (thus the job is available, see above) but the exporter is unable to contact the service for some reason. Such conditions are reported in metrics such as &lt;code&gt;mysql_up&lt;/code&gt; for example by the mysql exporter. See also https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets for dashboard and logs.

=== Failover Prometheus Pushgateway ===
The Prometheus Pushgateway needs to run as a singleton to properly track pushed metrics. For this reason the &lt;tt&gt;prometheus-pushgateway&lt;/tt&gt; is active on one host at a time.

To failover the steps involved are the following:

* Change &lt;tt&gt;profile::prometheus::pushgateway_host&lt;/tt&gt; in Puppet to point to another Prometheus host
* Change the &lt;tt&gt;prometheus-pushgateway.discovery.wmnet&lt;/tt&gt; record in DNS to point to the same host.
* Run puppet on the old and new host, then on all prometheus hosts in codfw/eqiad to make sure metrics are polled from the new host

=== Stale file for node-exporter textfile ===
Certain metrics are periodically generated by dumping Prometheus-formatted plaintext files (extension &lt;tt&gt;.prom&lt;/tt&gt;) into &lt;tt&gt;/var/lib/prometheus/node.d/&lt;/tt&gt;. The processes that generate the files run asynchronously to node-exporter, normally via systemd timers, and such processes can fail to update the files.  The alert fire whenever such metric files have failed to be updated; the Icinga alert description will be something like the following:

  cluster=analytics file=debian_version.prom instance=an-worker1101 job=node site=eqiad

Meaning that &lt;tt&gt;an-worker1101&lt;/tt&gt; has failed to update &lt;tt&gt;debian_version.prom&lt;/tt&gt;. Debugging such failures usually involved finding out which systemd timer is responsible for generating the file, usually by looking at puppet, and further debug from there.

Some prometheus prom.d metrics are not generated by systemd timers, but via confd. As such, they are not regularly rewritten but only if the underlying data changes. If the staleness alert triggers for such a metric it can be added to an ignore list in the alert, an example commit can be found at https://gerrit.wikimedia.org/r/c/operations/alerts/+/1201852

=== Pingthing Non-23xx HTTP response ===
A URL checked by the blackbox/pingthing prometheus job is returning a non 200/300 HTTP code, the URL contained in the instance label should be checked for problems.

=== Depool Prometheus for reads and writes ===
If the Prometheus host is in eqiad/codfw then depooling from the read path involves issuing a &lt;code&gt;depool&lt;/code&gt; and then &lt;code&gt;pool&lt;/code&gt; on the host itself (or via &lt;code&gt;confctl&lt;/code&gt;). For Prometheus on PoPs there's no depool on the read path.

For the &quot;write&quot; path (i.e. send alerts) the depool consists on setting &lt;code&gt;alertmanagers: []&lt;/code&gt; in hiera for the host in question and revert the change once done.

=== PrometheusRuleEvaluationFailures alert ===
This alert indicates that Prometheus or Thanos are failing to evaluate their recording or alerting rules. For further investigation check out the alert details on [https://alerts.wikimedia.org/?q=%40state%3Dactive&amp;q=%40cluster%3Dwikimedia.org&amp;q=alertname%3DPrometheusRuleEvaluationFailures alerts.w.o]. The cluster label will indicate prometheus or titan for prometheus and thanos respectively. Inspect the relevant systemd unit logs (e.g. &lt;code&gt;prometheus@k8s&lt;/code&gt; or &lt;code&gt;thanos-rule&lt;/code&gt;) to find out more about the error and which rule(s) are affected. See also https://phabricator.wikimedia.org/T387827 for an example investigation.

=== PrometheusDown ===
This alert fires whenever Prometheus is detected as down (i.e. Prometheus' self metrics can not be queried) by Thanos. The first and likely most effective solution is to reboot the affected &lt;code&gt;prometheus*&lt;/code&gt; hosts. Check out the linked dashboard from the alert to get an overview of which hosts are down/unresponsive.
[[Category:Metrics]]
[[Category:SRE Observability]]
[[Category:Services]]</text>
      <sha1>barauyq4uttaaprbmf0hdzsqgw4iusa</sha1>
    </revision>
  </page>
  <page>
    <title>User:Nhatminh01/Books/Deployments</title>
    <ns>2</ns>
    <id>442971</id>
    <revision>
      <id>2414317</id>
      <parentid>1805334</parentid>
      <timestamp>2026-05-16T19:57:29Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <origin>2414317</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="468" sha1="5ato656zsk6e3n8g0hijifz5snleiud" xml:space="preserve">:[[Cookbooks]]
:[[Cumin]]
:[[Deployments]]
:[[Deployments/Archive]]
:[[Deployments/Holding the train]]
:[[Dsh]]
:[[Heterogeneous deployment]]
:[[Heterogeneous deployment/Train deploys]]
:[[How to deploy code]]
:[[How to deploy current master branch of an extension]]
:[[How to deploy Wikidata code]]
:[[How to perform security fixes]]
:[[LocalisationUpdate]]
:[[Puppet]]
:[[Scap3]]
:[[Services]]
:[[Services/Deployment]]
:[[Services/FirstDeployment]]
:[[SWAT deploys]]</text>
      <sha1>5ato656zsk6e3n8g0hijifz5snleiud</sha1>
    </revision>
  </page>
  <page>
    <title>User:Nhatminh01/Books/Server admin log</title>
    <ns>2</ns>
    <id>442972</id>
    <revision>
      <id>2414318</id>
      <parentid>1805339</parentid>
      <timestamp>2026-05-16T19:57:37Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <origin>2414318</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1321" sha1="e45uq1exyahfcffwys9cu3gbfy5kxu7" xml:space="preserve">:[[Cloud VPS Server Admin Log]]
:[[Release Engineering/SAL]]
:[[Server Admin Log]]
:[[Server admin log/2008-08]]
:[[Server admin log/2008-09]]
:[[Server admin log/Archive 1]]
:[[Server admin log/Archive 10]]
:[[Server admin log/Archive 11]]
:[[Server admin log/Archive 12]]
:[[Server admin log/Archive 13]]
:[[Server admin log/Archive 14]]
:[[Server admin log/Archive 15]]
:[[Server admin log/Archive 16]]
:[[Server admin log/Archive 17]]
:[[Server admin log/Archive 18]]
:[[Server admin log/Archive 19]]
:[[Server admin log/Archive 2]]
:[[Server admin log/Archive 20]]
:[[Server admin log/Archive 21]]
:[[Server admin log/Archive 22]]
:[[Server admin log/Archive 23]]
:[[Server admin log/Archive 24]]
:[[Server admin log/Archive 25]]
:[[Server admin log/Archive 26]]
:[[Server admin log/Archive 27]]
:[[Server admin log/Archive 28]]
:[[Server admin log/Archive 29]]
:[[Server admin log/Archive 3]]
:[[Server admin log/Archive 30]]
:[[Server admin log/Archive 31]]
:[[Server admin log/Archive 32]]
:[[Server admin log/Archive 33]]
:[[Server admin log/Archive 34]]
:[[Server admin log/Archive 35]]
:[[Server admin log/Archive 36]]
:[[Server admin log/Archive 4]]
:[[Server admin log/Archive 5]]
:[[Server admin log/Archive 6]]
:[[Server admin log/Archive 7]]
:[[Server admin log/Archive 8]]
:[[Server admin log/Archive 9]]</text>
      <sha1>e45uq1exyahfcffwys9cu3gbfy5kxu7</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng/SAL</title>
    <ns>498</ns>
    <id>443628</id>
    <revision>
      <id>2414341</id>
      <parentid>2414014</parentid>
      <timestamp>2026-05-17T08:55:01Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25986406214 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)</comment>
      <origin>2414341</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="16641" sha1="km4ej3mgydaf5vtp60q473jtp0dgckw" xml:space="preserve">=== 2026-05-17 ===
* 08:55 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25986406214 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752565 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616046 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25727821548 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842111 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562306 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480578 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280492 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-07 ===
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25505647057 (https://github.com/cluebotng/component-configs/commits/874d7f6f407fc9a3995f52f40312cd7d3a712176)

=== 2026-05-06 ===
* 18:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754913 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-04 ===
* 07:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-05-02 ===
* 12:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25252392662 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278570 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)
* 00:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24865518502 (https://github.com/cluebotng/component-configs/commits/6e953f35fdca38226cde9ea7280f948f34242881)

=== 2026-04-23 ===
* 23:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24863962598 (https://github.com/cluebotng/component-configs/commits/07b9ad8616853af2ed49f96de6e55c14e3faabe0)
* 09:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24827343249 (https://github.com/cluebotng/component-configs/commits/218a7fa56222cf3e98b642eebbf6b1b2b273a92d)
* 09:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826859245 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79)
* 09:07 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826656498 (https://github.com/cluebotng/component-configs/commits/90aa39b3e81f19f42d08d5ec6131ba09c68bd786)

=== 2026-04-17 ===
* 00:12 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24540678933 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:05 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24422836823 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442412 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:15 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24249897963 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247614873 (https://github.com/cluebotng/component-configs/commits/30bda68a3ea7a1674d174e43cc8651d301c7485c)
* 14:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247570676 (https://github.com/cluebotng/component-configs/commits/93ced49392782bf65e34d13f10cbeaafa760f115)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093216 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656823 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659535101 (https://github.com/cluebotng/component-configs/commits/b825610c7f23870a8561b00b5f8546b107643015)

=== 2026-03-25 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23535729395 (https://github.com/cluebotng/component-configs/commits/c1468d960041cd66ab50902f344fec1ac65ddcad)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640340 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383557597 (https://github.com/cluebotng/component-configs/commits/0b67216b47074dd5d1d279dde0aa9144b243cf01)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551615 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23381633193 (https://github.com/cluebotng/component-configs/commits/ac40c461942f4541b640a32ff0141268418abc12)

=== 2026-03-19 ===
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23315406679 (https://github.com/cluebotng/component-configs/commits/fd07020c08545c83ab35667616a26081966648df)

=== 2026-02-15 ===
* 10:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22034380291 (https://github.com/cluebotng/component-configs/commits/842b50dc5d3160000352a25c5fdf09ea88ebf3eb)

=== 2025-11-11 ===
* 15:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642865 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270301650 (https://github.com/cluebotng/component-configs/commits/f28dcaec8c5882b4a1b7d861fe7f5e400312a5b4)
* 13:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266856634 (https://github.com/cluebotng/component-configs/commits/b0e9170597a778654185be762c580e2a6e19492f)

=== 2025-11-06 ===
* 01:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19121350626 (https://github.com/cluebotng/component-configs/commits/80f69f0ab7b09c2e3e5e208d847a954cb1975bc6)

=== 2025-11-05 ===
* 19:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19113995211 (https://github.com/cluebotng/component-configs/commits/586f2c46dcbbb09a9f7926e991bc5fbe45f4a1e9)
* 18:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19112506240 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)
* 16:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108852168 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)
* 12:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19102217917 (https://github.com/cluebotng/component-configs/commits/24f3dc9fe5e2211d861c754a4b9342a6127f4a4a)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872623 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267293 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-08 ===
* 15:47 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.1
* 11:23 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.4.0

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101448 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 09:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092350259 (https://github.com/cluebotng/component-configs/commits/283965c9240c0c5a72e0ea1203439583935295cb)

=== 2025-09-27 ===
* 13:07 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.3.0

=== 2025-09-26 ===
* 18:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649036 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca)
* 12:52 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.4
* 12:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18038096319 (https://github.com/cluebotng/component-configs/commits/e10e601b4fb06b3fd97856ef86a30e5391fb4f17)
* 12:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926507 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d)
* 12:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037156332 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)

=== 2025-09-25 ===
* 21:20 wmbot~damian@tools-bastion-15: bot deployed @ v1.2.2
* 21:20 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.2
* 19:17 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.1
* 19:16 wmbot~damian-scripts@tools-bastion-15: bot deployed @ refs/tags/v1.2.0
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18017003184 (https://github.com/cluebotng/component-configs/commits/e9fc8d46ac1a0ff0ac6203458fa171c6430492ce)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016894528 (https://github.com/cluebotng/component-configs/commits/cfc9adc9516df0f11c8b6d1df68232d0a46cb4eb)
* 17:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998407 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 17:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 17:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801881 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221965 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782359 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677281 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531091 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)

=== 2025-09-22 ===
* 19:08 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.3
* 19:04 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.2
* 19:02 wmbot~damian-scripts@tools-bastion-15: report deployed @ refs/tags/1.2.1

=== 2025-08-29 ===
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.2.0
* 14:30 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/heads/main
* 00:10 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.1

=== 2025-08-15 ===
* 21:14 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.1.0
* 20:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/1.0.34
* 12:58 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.33
* 00:23 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.32
* 00:08 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.31

=== 2025-08-14 ===
* 16:13 wmbot~damian@tools-bastion-13: irc-relay deployed @ v1.1.12
* 12:47 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2
* 12:43 wmbot~damian@tools-bastion-13: core deployed @ v0.0.2

=== 2025-08-11 ===
* 13:06 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.3
* 12:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.30

=== 2025-08-10 ===
* 17:42 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.2
* 17:37 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.29
* 16:35 wmbot~damian-scripts@tools-bastion-13: bot deployed @ refs/tags/v1.1.1

=== 2025-08-08 ===
* 15:37 wmbot~damian@tools-bastion-12: Updated ci-execute-fabric to use dedicated unix account, dropped key from human account
* 15:36 wmbot~damian-scripts@tools-bastion-13: report deployed @ refs/tags/v1.0.28
* 15:33 wmbot~damian@tools-bastion-13: report deployed @ refs/tags/v1.0.28

=== 2025-08-07 ===
* 15:24 wmbot~damian@tools-bastion-13: report deployed @ v1.0.27

=== 2024-03-21 ===
* 10:13 dcaro: fixed .lighttpd.conf file to add port and remove socket

=== 2021-07-23 ===
* 16:40 majavah: stop cbng_relay grid job, still having issues with irc connection - [[phab:T274871|T274871]]

=== 2020-01-13 ===
* 20:53 wm-bot: &lt;root&gt; Restarted webservice to fix broken registration with the front proxy ([[phab:T242538|T242538]])

=== 2019-03-07 ===
* 05:06 bd808: Killed cbng_bot job stuck in deletion state with 4000+ zombie child processes ([[phab:T217817|T217817]])


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>km4ej3mgydaf5vtp60q473jtp0dgckw</sha1>
    </revision>
  </page>
  <page>
    <title>Docker-registry/Runbook</title>
    <ns>0</ns>
    <id>443852</id>
    <revision>
      <id>2414325</id>
      <parentid>2402816</parentid>
      <timestamp>2026-05-16T22:10:19Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414325</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="6616" sha1="65kz4y173cqk0a2qr3rejhqig9beeyr" xml:space="preserve">{{Draft}}

==On-call guide==

===Swift is down or content has gone corrupt===
If swift content has gone the registry would not be capable of serving images or writing new ones, you need to focus first on Swift outage and when resolved check if content is still there or you need to rebuild and republish every image. One thing it could be done to mitigate at some extent this failure is cache the most frequent content of the registry on the caching layer (varnish or ATS) that would allow for pulling the most used images.

===Redis is down===
Redis is only used as a blob cache, if it is down pulling and pushing images would be slower but registry will continue to serve content. 

===Registry is down in one DC, how to failover to the passive?===
Use common pool/depool operations. Take in account that clients must use the discovery DNS entry and that the swift replication is usually slow, which means that if there was an image pushed just before the outage this image would probably have not been replicated yet to the other DC.

Users should trigger a rebuild/repush or cache locally the image to avoid issues.

=== Cannot pull some layer/image or looks corrupt how to debug problems? ===

==== Check swift replication ====

* See relevant section.

==== Replication seems synced but users/monitoring reports problems pulling images ====

* checkout registry logs, it should report on errors which image/manifest/tag is failing 
* Increase the loglevel of the registry to debug, modify registry configuration on /etc/docker/registry/config.yml and include the following snippet under the log key. After that restart the registry with systemctl restart docker-registry
&lt;syntaxhighlight lang=&quot;text&quot;&gt;
log:
  level: debug
&lt;/syntaxhighlight&gt;

* Registry will log which swift or redis calls are failing and after that you will need to dig more in swift or redis. In the event of a swift failure usually republishing images will help 

=== The replication between Swift codfw and eqiad is not working. ===
The first sanity check is to execute the following:

*Ssh in a registry on eqiad and run as root source &lt;code&gt;/etc/swift/account_docker\:registry.env&lt;/code&gt; &amp;&amp; &lt;code&gt;swift stat docker_registry_codfw&lt;/code&gt;
*Ssh in a registry on codfw and run as root source &lt;code&gt;/etc/swift/account_docker\:registry.env&lt;/code&gt; &amp;&amp; &lt;code&gt;swift stat docker_registry_codfw&lt;/code&gt;
* Compare number of objects and size, replication is slow specially if there has been a lot of activity in a spike. Check also Swift metrics in the [https://grafana.wikimedia.org/d/StcefURWz/docker-registry docker-registry's dashboard].

If the numbers are not in sync, there is a problem. The Docker Registry is currently (Dec 2025) the only Swift client using the native container replication settings, namely it is Swift itself taking care of replicating the objects. Every Swift host runs a service called &lt;code&gt;swift-container-sync&lt;/code&gt;, but only three of them take care of a specific container replication. In our case, since the replication is from codfw to eqiad, we should verify if any error is listed in those hosts. To quickly find them, you can run something like the following from cumin hosts:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
sudo cumin 'ms-be2*' 'systemctl status swift-container-sync.service' -b 10 -p 90
&lt;/syntaxhighlight&gt;The last problem registered was https://phabricator.wikimedia.org/T413008, where for some reason that we didn't get the sqllite databases used by the container-sync daemons got inconsistent between the three replicas, and we had to issue multiple swift DELETE requests to force Swift to realign the three replicas (more info in the task, the procedure is probably a one-off). If this appears to be the problem, then it's best to reach out to the Swift service owners in SRE Data Persistence (e.g. by opening a Phabricator task tagged #sre-swift-storage)

==FAQ==

===I need to delete a tag of a published image===
This is not currently possible in the V2 Registry API, you should republish the tag if the content is wrong or ignore it.

===I need to delete an image from the registry===
You need to ssh in any registry instance and delete the objects that belongs to the image from the swift container, this should not be done unless there are good reasons to do it (security incident for instance).

===How to perform garbage collection===
Registry will have more layers than the one referenced from manifests, if you want to delete orphan blobs just log in registry instances and execute &lt;code&gt;/usr/bin/docker-registry garbage-collect /etc/docker/registry/config.yml&lt;/code&gt;

=== How to backup the backend swift container ===
If you need to modify the underlying swift container you may want to backup the content before modifying replication options or other things. This is an extremely dangerous thing and you should do it as a last resort.  when the registry writes to swift it writes two kinds of objects 'files' and 'segments' they need to be backed up differently due to segments having a extremely long filename that will make swift cli crash using download/upload.

* ssh into a server that has swift docker credentials and a los of free disk space, swift frontends are usually a good choice (ms-fe*)
* Create a directory under /tmp like /tmp/backup
* Move to /tmp/backup and run &lt;code&gt;source /etc/swift/account_AUTH_docker.env; ionice -c3 swift download --skip-identical -p prefix/ --object-threads 3 --container-threads 3 SWIFT_CONTAINER.&lt;/code&gt;The swift download command will start to download to disk the content of 'files' prefix. You can run this multiple times to get updates.
* When the download has finished, you should execute &lt;code&gt;swift upload --skip-identical -c --object-threads 3 --segment-threads 3 SWIFT_BACKUP_CONTAINER files/.You can run this command multiple times&lt;/code&gt;
* For segments something different needs to be done as segments has an extremely long filename that will make swift cli crash. cd into /tmp/backup and launch &lt;code&gt;source /etc/swift/account_AUTH_docker.env;swift list -p segments SWIFT_SOURCE_CONTAINER &gt; /tmp/filesa; swift list -p SWIFT_BACKUP_CONTAINER &gt; /tmp/filesb;comm -3 &lt;(sort /tmp/filesa) &lt;(sort /tmp/filesb) | xargs -I '{}' -s 300000 bash -c &quot;swift copy -d /SWIFT_BACKUP_CONTAINER SWIFT_SOURCE_CONTAINER '{}' &amp;&amp; date&quot;.&lt;/code&gt; You can run this command multiple times to get updates.

==Known problems==

==== Aggressive APUS caching ====
When the [[Ceph/Cephadm|APUS service]] changes pooled status, docker-registry services need to be restarted. Instructions for this process are located at [[Ceph/Cephadm#Pooling / Depooling]].
[[Category:Runbooks]]</text>
      <sha1>65kz4y173cqk0a2qr3rejhqig9beeyr</sha1>
    </revision>
  </page>
  <page>
    <title>Portal:Cloud VPS/Admin/Ceph</title>
    <ns>0</ns>
    <id>445092</id>
    <revision>
      <id>2414326</id>
      <parentid>2399054</parentid>
      <timestamp>2026-05-16T22:11:36Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414326</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="48947" sha1="ab3nj0289rmn2y6e47zrptcn529gq59" xml:space="preserve">'''[[w:Ceph (software)|Ceph]]''' is a distributed storage platform which is used to provide shared block storage services for Cloud VPS instance disk and operating system images. This is a low level support system which is interesting to the [[Wikimedia Cloud Services team]] and others who work on the OpenStack hosting platform for the Cloud VPS product. It is not currently a Cloud VPS project admin or user facing feature.

== Architecture ==
=== Hardware sizing ===
Hardware sizing recommendations using the results from our internal testing and community guidelines:

{| class=&quot;wikitable&quot;
|-
! scope=&quot;col&quot;| Type
! scope=&quot;col&quot;| Bus
! scope=&quot;col&quot;| Bandwidth (AVG)
! scope=&quot;col&quot;| CPU Recommendations
! scope=&quot;col&quot;| RAM Recommendations
|-
! scope=&quot;row&quot;| HDD
| SATA 6Gb/s
| 150MB/s
| Xeon Silver 2GHz + 1 cpu core = 2 core-GHz per HDD
| 1GB for 1TB of storage
|-
! scope=&quot;row&quot;| SSD'''*'''
| SATA 6Gb/s
| 450MB/s
| Xeon Silver 2GHz + 2 cpu cores = 4 core-GHz per SSD
| 1GB for 1TB of storage
|-
! scope=&quot;row&quot;| NVME
| M.2. PCIe 32 Gb/s
| 3000MB/s
| Xeon Gold 2GHz * 5 cpu cores * 2 sockets = 20 core-GHz per NVME
| 2GB for each OSD
|}
'''* POC cluster is equipped with SATA SSD drives'''

''Note: When reviewing the sizing guides in the community keep in mind the types of drives and their capabilities.''

In addition to the baseline CPU requirements, it's necessary to include additional CPU and RAM for the operating system and Ceph rebuilding, rebalancing and data scrubbing.

Next Phase Ceph OSD Server recommendation:

PowerEdge R440 Rack Server
* 2 x Xeon Silver 4214 CPU 12 cores / 24 threads
* 2 x 32GB RDIMM
* 2 x 240GB SSD SATA (OS Drive)
* 8 x 1.92TB SSD SATA (Data Drive)
* 2 x 10GB NIC
* No RAID ([https://en.wikipedia.org/wiki/Non-RAID_drive_architectures#JBOD JBOD] Only)

It's estimated that 15 OSD servers will provide enough '''initial''' storage capacity for existing virtual machine disk images and block devices.

=== BIOS ===
==== Base Settings ====

* Ensure &quot;Boot mode&quot; is set to &quot;BIOS&quot; and not &quot;UEFI&quot;. (This is required for the netboot process)

==== PXE boot settings ====
All Ceph hosts are equipped with a 10Gb Broadcom BCM57412 NIC and are not using the embedded onboard NIC. 

# During the system boot, when prompted to configure the second Broadcom BCM57412 NIC device press &quot;ctrl + s&quot;
# On the main menu select &quot;MBA Configuration&quot; and toggle the &quot;Boot Protocol&quot; setting to &quot;Preboot Execution Environment (PXE)&quot;
# Press escape, then select &quot;Exit and Save Configurations&quot;
# After the system reboots, press &quot;F2&quot; to enter &quot;System Setup&quot;
# Navigate to &quot;System BIOS &gt; Boot Settings &gt; BIOS Boot Settings&quot;
# Select &quot;Boot Sequence&quot; and change the boot order to: &quot;Hard dive C:&quot;, &quot;NIC in Slot 2 Port 1..&quot;, &quot;Embedded NIC 1 Port 1...&quot;
# Exit System Setup, saving your changes and rebooting the system

Alternatively steps 4 through 7 can be replaced with racadm, but you will still need to enable the PXE boot protocol in the option ROM. 
 /admin1-&gt; racadm set BIOS.BiosBootSettings.bootseq HardDisk.List.1-1,NIC.Slot.2-1-1,NIC.Embedded.1-1-1
 /admin1-&gt; racadm jobqueue create BIOS.Setup.1-1
 /admin1-&gt; racadm serveraction hardreset

=== Storage ===
==== Operating System Disks ====
The Ceph OSD and Monitor servers are both using software mdadm RAID1 for the operating system. This RAID set is built using the first 2 (sda and sdb) SSD drives with the partman/raid1-2dev.cfg profile. 

When mixing RAID and NON-RAID devices with the LSI Logic / Symbios Logic MegaRAID SAS-3 3108 storage adapter the device IDs get mapped incorrectly for our environment. The NON-RAID devices show up first followed by the RAID device. Due to this we've selected to use software MDADM RAID instead of hardware RAID.

==== Ceph OSD Disks ====
Ceph manages data redundancy in the application layer, because of this each OSD drive is configured as a JBOD in NON-RAID mode. 

==== SSD Drive Details ====
{| class=&quot;wikitable&quot;
|-
! scope=&quot;col&quot;| Server
! scope=&quot;col&quot;| Total
! scope=&quot;col&quot;| Vendor
! scope=&quot;col&quot;| Capacity
! scope=&quot;col&quot;| Model
! scope=&quot;col&quot;| Function
|-
! scope=&quot;row&quot;| cloudcephmon100[1-3].wikimedia.org
| 2
| Dell
| 480GB
| SSDSC2KB480G8R
| RAID1 operating system
|-
! scope=&quot;row&quot;| cloudcephosd100[1-3].wikimedia.org
| 2
| Dell
| 240GB
| SSDSC2KG240G8R
| RAID1 operating system
|-
! scope=&quot;row&quot;| cloudcephosd100[1-3].wikimedia.org
| 8
| Dell
| 1.92TB
| SSDSC2KG019T8R
| JBOD Ceph Bluestore
|-
|}

[[File:CloudVPS-ceph-disk-layout-1.png]]

=== Network ===

Description and configuration hints about the network as related to the ceph project.  Note that our ceph cluster is ipv4 only.  Ceph can only do one or the other, and ipv4 is more useful for our current cloud setup. 

==== switches ====

[[File:Cloudsw.png|thumb]]

Given lack of 10G ports and physical racking space in eqiad row B, we got new dedicated 10G switches called '''cloudsw''' in different rows D and C.

There are physical cables that connect '''cloudsw1-d5-eqiad''' and '''cloudsw1-c8-eqiad''', and then '''cloudsw1-c8-eqiad''' to '''asw2-b2''' and '''asw2-b7'''.

Both '''cloudsw* and asw2-b*''' devices have at least the following 2 VLANs defined and trunked:
* the special storage VLAN/subnet used for inter-cluster sync [https://netbox.wikimedia.org/ipam/vlans/142/ cloud-storage1-eqiad - vlan 1106] with addressing 192.168.x.y/z
* the SSH management and client traffic VLAN/subnet [https://netbox.wikimedia.org/ipam/vlans/88/ cloud-hosts1-b-eqiad - vlan 1118] with addressing 10.64.20.0/24 

The special storage VLAN/subnet doesn't have a L3 gateway configured anywhere.

This is, we are stretching VLANs across rows.

==== servers ====

[[File:Cloudvps-ceph-phase1-2.png|right|thumb]]

Ceph storage nodes ('''cloudcephosdXXXX''') have 2 NICs connected to 2 different networks:
* eth0: [https://netbox.wikimedia.org/ipam/vlans/88/ cloud-hosts1-b-eqiad - vlan 1118] with addressing 10.64.20.0/24 -- used for SSH management and client traffic. 
* eth1: [https://netbox.wikimedia.org/ipam/vlans/142/ cloud-storage1-eqiad - vlan 1106] with addressing 192.168.x.y/z -- used for inter-cluster synchronization.

Ceph monitor nodes ('''cloudcephmonXXXX''') have 1 NIC connected to:
* eth0: [https://netbox.wikimedia.org/ipam/vlans/88/ cloud-hosts1-b-eqiad - vlan 1118] with addressing 10.64.20.0/24 -- used for SSH management and client traffic.

Hypervisor servers ('''cloudvirtXXXX''') are expected to connect to '''cloudcephmonXXXX''' using the '''cloud-hosts1-b-eqiad''' subnet, thus a direct connection (no L3 gateway involved).

== Operating System ==

All Ceph hosts will be using Debian 10 (Buster). 

=== Ceph Packaging ===
Buster does not include a recent version of Ceph and Ceph does not provide packages for Buster (tracked at https://tracker.ceph.com/issues/42870). 

Croit provides Debian Buster packages that are built using the Ceph build process and available to the community https://croit.io/2019/07/07/2019-07-07-debian-mirror https://github.com/croit/ceph. These packages have been mirrored to our local APT repository.

== Puppet ==

=== Roles ===
* '''wmcs::ceph::mon''' Deploys the Ceph monitor and manager daemon to support CloudVPS hypervisors
* '''wmcs::ceph::osd''' Deploys the Ceph object storage daemon to support CloudVPS hypervisors
* '''role::wmcs::openstack::eqiad1::virt_ceph''' Deploys nova-compute configured with RBD based virtual machines (Note that switching between this role and wmcs::openstack::eqiad1::virt should work gracefully without a rebuild)

=== Profiles ===
* '''profile::ceph::client::rbd''' Install and configure a Ceph RBD client
* '''profile::ceph::osd''' Install and configure Ceph object storage daemon
* '''profile::ceph::mon''' Install and configure Ceph monitor and manager daemon
* '''profile::ceph::alerts''' Configure Ceph cluster alerts in Icinga

=== Modules ===
* '''ceph''' Install and configure the base Ceph installation used by all services and clients
* '''ceph::admin''' Configures the Ceph administrator keyring
* '''ceph::mgr''' Install and configure the Ceph manager daemon
* '''ceph::mon''' Install and configure the Ceph monitor daemon
* '''ceph::keyring''' Defined resource that manages access control and keyrings

== Post Installation Procedures ==

=== Adding OSDs ===
==== Using cookbooks ====
===== Pre-flight checks =====
You have to make sure that the following is setup correctly:
* Add the host to the site.pp puppet file (so it gets the &lt;code&gt;wmcs::ceph::osd&lt;/code&gt; role).
* Add the host info to the hiera variable &lt;code&gt;profile::ceph::osd::hosts&lt;/code&gt;(currently under &lt;code&gt;operations/puppet&lt;/code&gt; repo, for example &lt;code&gt;hieradata/eqiad/profile/ceph/osd.yaml&lt;/code&gt;), note that you'll have to add it to the section according to the rack it's on, and manually assign the next free ip for the internal interface.
* On netbox, you have to manually add an entry to the ip block for the cloud-private network according to it's rack:
** c8 - https://netbox.wikimedia.org/ipam/prefixes/653/
** d5 - https://netbox.wikimedia.org/ipam/prefixes/654/
** e4 - https://netbox.wikimedia.org/ipam/prefixes/655/
** f4 - https://netbox.wikimedia.org/ipam/prefixes/656/

===== Run the cookbook =====
Once that is merged, you can run the [[Wikimedia Cloud Services team/EnhancementProposals/Operational Automation|cookbook]], from your laptop or from the &lt;code&gt;cloudcumin&lt;/code&gt; node:
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
dcaro@vulcanus$ cookbook wmcs.ceph.osd.bootstrap_and_add -h
usage: cookbook [GLOBAL_ARGS] wmcs.ceph.osd.bootstrap_and_add [-h] [--project PROJECT] [--task-id TASK_ID] [--no-dologmsg] --cluster-name {eqiad1,codfw1} --osd-hostname OSD_HOSTNAME [--skip-reboot] [--only-check] [--yes-i-know-what-im-doing] [--batch-size BATCH_SIZE] [--wait-for-rebalance] [--force]

WMCS Ceph - Bootstrap a new osd

Usage example:
    cookbook wmcs.ceph.osd.bootstrap_and_add \
        --new-osd-fqdn cloudcephosd1016.eqiad.wmnet \
        --task-id T12345

options:
  -h, --help            show this help message and exit
  --project PROJECT     Relevant Cloud VPS openstack project (for operations, dologmsg, etc). If this cookbook is for hardware, this only affects dologmsg calls. Default is 'admin'.
  --task-id TASK_ID     Id of the task related to this operation (ex. T123456). (default: None)
  --no-dologmsg         To disable dologmsg calls (no SAL messages on IRC). (default: False)
  --cluster-name {eqiad1,codfw1}
                        Ceph cluster to roll restart. (default: None)
  --osd-hostname OSD_HOSTNAME
                        Hostname of the new OSDs to add. Repeat for each new OSD. If specifying more than one, consider passing --yes-i-know-what-im-doing (default: None)
  --skip-reboot         If passed, will not do the first reboot before adding the new osds. Useful when the machine has already some running OSDs and you are sure the reboot is not needed. (default: False)
  --only-check          If passed, will only run the pre-setup checks on the host and report back, nothing more. (default: False)
  --yes-i-know-what-im-doing
                        If passed, will not ask for confirmation. WARNING: this might cause data loss, use only when you are sure what you are doing. (default: False)
  --batch-size BATCH_SIZE
                        Number of osds to bring up at a time to avoid congesting the network, use 0 for all at once. (default: 4)
  --wait-for-rebalance  If passed, will wait for the cluster to do the rebalancing after adding the new OSDs. Note that this might take several hours. (default: False)
  --force               If passed, will continue even if the cluster is not in a healthy state. (default: False)
&lt;/syntaxhighlight&gt;

An example run would be:
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
cookbook wmcs.ceph.osd.bootstrap_and_add --cluster-name eqiad1 --task-id T371878 --osd-hostname cloudcephosd1035 --wait-for-rebalance
&lt;/syntaxhighlight&gt;

==== Manually ====
Ensure OSD nodes can talk to Ceph nodes:

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
root@cloudcephmon2001-dev: ceph auth import -i /var/lib/ceph/bootstrap-osd/ceph.keyring
&lt;/syntaxhighlight&gt;

Locate available disks with lsblk
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
 cloudcephosd1001:~# lsblk
 NAME                                                                                                  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
 sda                                                                                                     8:0    0 223.6G  0 disk
 ├─sda1                                                                                                  8:1    0  46.6G  0 part
 │ └─md0                                                                                                 9:0    0  46.5G  0 raid1 /
 ├─sda2                                                                                                  8:2    0   954M  0 part
 │ └─md1                                                                                                 9:1    0   953M  0 raid1 [SWAP]
 └─sda3                                                                                                  8:3    0 176.1G  0 part
   └─md2                                                                                                 9:2    0   176G  0 raid1
     └─cloudcephosd1001--vg-data                                                                       253:2    0 140.8G  0 lvm   /srv
 sdb                                                                                                     8:16   0 223.6G  0 disk
 ├─sdb1                                                                                                  8:17   0  46.6G  0 part
 │ └─md0                                                                                                 9:0    0  46.5G  0 raid1 /
 ├─sdb2                                                                                                  8:18   0   954M  0 part
 │ └─md1                                                                                                 9:1    0   953M  0 raid1 [SWAP]
 └─sdb3                                                                                                  8:19   0 176.1G  0 part
   └─md2                                                                                                 9:2    0   176G  0 raid1
     └─cloudcephosd1001--vg-data                                                                       253:2    0 140.8G  0 lvm   /srv
 sdc                                                                                                     8:80   0   1.8T  0 disk
 sdd                                                                                                     8:96   0   1.8T  0 disk
 sde                                                                                                     8:80   0   1.8T  0 disk
 sdf                                                                                                     8:80   0   1.8T  0 disk
 sdg                                                                                                     8:96   0   1.8T  0 disk
 sdh                                                                                                     8:112  0   1.8T  0 disk
 sdi                                                                                                     8:128  0   1.8T  0 disk
 sdj                                                                                                     8:144  0   1.8T  0 disk
&lt;/syntaxhighlight&gt;

To prepare a disk for Ceph first zap the disk
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
 cloudcephosd1001:~# ceph-volume lvm zap /dev/sdc
 --&gt; Zapping: /dev/sdc
 --&gt; --destroy was not specified, but zapping a whole device will remove the partition table
 Running command: /bin/dd if=/dev/zero of=/dev/sdc bs=1M count=10
  stderr: 10+0 records in
 10+0 records out
 10485760 bytes (10 MB, 10 MiB) copied, 0.00357845 s, 2.9 GB/s
 --&gt; Zapping successful for: &lt;Raw Device: /dev/sdc&gt;
&lt;/syntaxhighlight&gt;

Then prepare, activate and start the new OSD
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
 cloudcephosd1001:~# ceph-volume lvm create --bluestore --data /dev/sde
&lt;/syntaxhighlight&gt;

You may need to explicitly set the osd class to SSD.  First check how it displays with
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
 ceph osd tree
&lt;/syntaxhighlight&gt;

If incorrect, set to ssd with:

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
ceph osd crush rm-device-class      osd.&lt;number&gt;
ceph osd crush set-device-class ssd osd.&lt;number&gt;
&lt;/syntaxhighlight&gt;

==== Common issues ====
===== OSDs not joining due to missed heartbeats =====
This happened only once, but it was tricky to figure out.
The behavior is that newly added osds start joining the cluster, but fail to reply/receive heartbeats from other osds and end up getting flagged out by the mon nodes.

On the osd logs you could see messages like:

&lt;pre&gt;
&gt;$ journalctl -f -u ceph-osd@191.service
...
Aug 17 11:03:37 cloudcephosd1025 ceph-osd[15007]: 2022-08-17 11:03:37.177 7f3559652700 -1 osd.191 17192523 heartbeat_check: no reply from 10.64.20.58:6826 osd.0 ever on either front or back, first ping sent 2022-08-17 11:03:15.776557 (oldest deadline 2022-08-17 11:03:35.776557)
Aug 17 11:03:38 cloudcephosd1025 ceph-osd[15007]: 2022-08-17 11:03:38.133 7f3559652700 -1 osd.191 17192523 heartbeat_check: no reply from 10.64.20.58:6826 osd.0 ever on either front or back, first ping sent 2022-08-17 11:03:15.776557 (oldest deadline 2022-08-17 11:03:35.776557)
Aug 17 11:03:38 cloudcephosd1025 ceph-osd[15007]: 2022-08-17 11:03:38.133 7f3559652700 -1 osd.191 17192523 heartbeat_check: no reply from 10.64.20.58:6826 osd.0 ever on either front or back, first ping sent 2022-08-17 11:03:15.776557 (oldest deadline 2022-08-17 11:03:35.776557)
...
&lt;/pre&gt;

The issue this time was that the jumbo frames were not allowed by the network gear on some routes ({{PhabT|315446}}), a quick way of checking could be:
&lt;pre&gt;
&gt;$ for ip in $(grep addr /etc/ceph/ceph.conf | awk '{print $4}'); do ping -M do -4 -c 1 -W 1 -s 8972 $ip 1&gt;/dev/null 2&gt;&amp;1 &amp;&amp; echo &quot;$ip: ok&quot; || echo &quot;$ip: nook&quot;; done
&lt;/pre&gt;

=== Creating Pools ===

To create a new storage pool you will first need to determine the number of placement groups that will be assigned to the new pool. You can use the calculator at https://ceph.io/pgcalc/ to help identify the starting point (not you can easily increase, but not decrease this value):

 sudo ceph osd pool create eqiad1-compute 512

Enable the RBD application for the new pool

 sudo ceph osd pool application enable eqiad1-compute rbd

=== Configuring Client Keyrings ===

The Cloud VPS hypervisors connect to the compute pool using RBD. After running the following command the keyring data should be stored in private hiera file: eqiad/profile/ceph/client/rbd.yaml with profile::ceph::client::rbd::keydata key. (Note all nova-compute hypervisors are configured to use the same Ceph keyring)

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
 cloudcephmon1001:~$ sudo ceph auth get-or-create client.eqiad1-compute mon 'profile rbd' osd 'profile rbd pool=compute'
&lt;/syntaxhighlight&gt;

As of 2021-02-23 many of the caps are managed by hand rather than puppetized.  To see the current state use

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
 cloudcephmon1001:~$ ceph auth ls
&lt;/syntaxhighlight&gt;

To adjust caps, use 'ceph auth caps'.  Here's an example of adding r/w rbd access to one pool and r/o access to another pool to the client.eqiad1-compute keyring:

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
root@cloudcephmon1001:~# ceph auth caps client.eqiad1-compute mon 'profile rbd' osd 'profile rbd-read-only pool=eqiad1-glance-images, profile rbd pool=eqiad1-compute'
updated caps for client.eqiad1-compute
&lt;/syntaxhighlight&gt;

== Monitoring ==

The Ceph managers are configured to run a [https://docs.ceph.com/docs/master/mgr/prometheus/ Prometheus exporter] that exports global cluster metrics. This exporter is only active on one manager at a time, failing over between active managers is handled directly by Ceph. The cloudmetrics Prometheus server is configured to scrape this exporter.

=== Logs ===
You can find all related logs here:
* [https://logstash.wikimedia.org/app/dashboards#/view/f7bd36a0-bedb-11eb-81e9-e1226573bad4?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-12h%2Cto%3Anow)) Codfw1 cluster]
* [https://logstash.wikimedia.org/app/dashboards#/view/a3dae590-bed7-11eb-85b7-9d1831ce7631?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-12h%2Cto%3Anow)) Eqiad1 cluster]
=== Dashboards ===
You can find [https://grafana-rw.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?layout=folders&amp;tag=ceph&amp;tag=WMCS&amp;query=folder:current%20ceph&amp;search=open&amp;folder=current&amp;orgId=1&amp;forceLogin=true all the ceph related dashboards].

=== Icinga alerts ===
These alerts are global to the Ceph cluster and configured on the icinga1001 host to avoid getting multiple notifications for a single event.

==== Ceph Cluster Health ====
*; Description : Ceph storage cluster health check

*; Status Codes
*:* 0 - healthy, all services are healthy
*:* 1 - warn, cluster is running in a degraded state, data is still accessible
*:* 2 - critical, cluster is failed, some or all data is inaccessible

*; Next steps
*:* On one of the ceph monitor hosts (e.g. cloudcephmon1001.wikimedia.org) check the output of the command &lt;code&gt;sudo ceph --status&lt;/code&gt;. Example output from a healthy cluster:
 cloudcephmon1001:~$ sudo ceph --status
  cluster:
    id:     5917e6d9-06a0-4928-827a-f489384975b1
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 (age 3w)
    mgr: cloudcephmon1002(active, since 10d), standbys: cloudcephmon1003, cloudcephmon1001
    osd: 24 osds: 24 up (since 3w), 24 in (since 3w)
 
  data:
    pools:   1 pools, 256 pgs
    objects: 3 objects, 19 B
    usage:   25 GiB used, 42 TiB / 42 TiB avail
    pgs:     256 active+clean

''Note: As of 2020-09-10, ceph health checks will sometimes incorrectly report a small number of slow ops. Restarting the active monitor service will clean up these warnings -- that's a good thing to try before obsessing over a slow ops warning.''

*; References
*:* https://docs.ceph.com/docs/master/rados/operations/monitoring/#monitoring-health-checks
*:* https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/EOWNO3MDYRUZKAK6RMQBQ5WBPQNLHOPV/
----

==== Ceph Monitor Quorum ====
*; Description : Verify there are enough Ceph monitor daemons running for proper quorum

*; Status Codes
*:* 0 - healthy, 3 or more Ceph Monitors are running
*:* 2 - critical, Less than 3 Ceph monitors are running

*; Next steps
*:* On one of the ceph monitor hosts (e.g. cloudcephmon1001.wikimedia.org) check the output of the command &lt;code&gt;sudo ceph mon stat&lt;/code&gt;. Example output from a healthy cluster:
 cloudcephmon1001:~$ sudo ceph mon stat
 e1: 3 mons at {cloudcephmon1001=[v2:208.80.154.148:3300/0,v1:208.80.154.148:6789/0],cloudcephmon1002=[v2:208.80.154.149:3300/0,v1:208.80.154.149:6789/0],cloudcephmon1003=[v2:208.80.154.150:3300/0,v1:208.80.154.150:6789/0]}, election epoch 24, leader 0 cloudcephmon1001, quorum 0,1,2 cloudcephmon1001,cloudcephmon1002,cloudcephmon1003

*; References
*:* https://docs.ceph.com/docs/master/rados/operations/monitoring/#checking-monitor-status

== Maintenance ==
=== Restarting Ceph nodes ===
You can use the cookbooks for roll rebooting ceph node types, they will take care of making sure the cluster is healthy at every step, silencing alert during the reboot, etc.:
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
dcaro@vulcanus$ cookbook -l wmcs.ceph.roll
cookbooks
`-- wmcs
    `-- wmcs.ceph
        |-- wmcs.ceph.roll_reboot_mons
        `-- wmcs.ceph.roll_reboot_osds
&lt;/syntaxhighlight&gt;


=== Restarting Ceph services ===
Always ensure the cluster is healthy before restarting any services

&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cloudcephosd1001:~$ sudo ceph -s
  cluster:
    id:     5917e6d9-06a0-4928-827a-f489384975b1
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 (age 14m)
    mgr: cloudcephmon1002(active, since 17m), standbys: cloudcephmon1001, cloudcephmon1003
    osd: 24 osds: 24 up (since 10w), 24 in (since 10w)
 
  data:
    pools:   1 pools, 256 pgs
    objects: 41.22k objects, 160 GiB
    usage:   502 GiB used, 41 TiB / 42 TiB avail
    pgs:     256 active+clean
 
  io:
    client:   25 KiB/s wr, 0 op/s rd, 1 op/s wr
&lt;/syntaxhighlight&gt;

==== ceph-crash ====
The ceph-crash service can be restarted on all hosts without any service interruption
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cumin1001:~$ sudo cumin 'P{R:Class = role::wmcs::ceph::mon or R:Class = role::wmcs::ceph::osd}' 'systemctl restart ceph-crash'
[..]
&lt;/syntaxhighlight&gt;

==== ceph-mgr ====
The ceph-mgr service can be restarted one by one across the ceph-mon hosts
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cumin1001:~$ sudo cumin 'cloudcephmon1001.wikimedia.org' 'systemctl restart ceph-mgr@cloudcephmon1001'
user@cumin1001:~$ sudo cumin 'cloudcephmon1002.wikimedia.org' 'systemctl restart ceph-mgr@cloudcephmon1002'
user@cumin1001:~$ sudo cumin 'cloudcephmon1003.wikimedia.org' 'systemctl restart ceph-mgr@cloudcephmon1003'
&lt;/syntaxhighlight&gt;

==== ceph-mon ====
&lt;span style=&quot;background:#ff0000 &quot;&gt;IMPORTANT: This process should be run by the WMCS team&lt;/span&gt;

&lt;span style=&quot;background:#ff0000 &quot;&gt;IMPORTANT: Only one Ceph-mon can be offline at anytime.&lt;/span&gt;

The ceph-mon service can be restarted one by one across the ceph-mon hosts, in between restarts ensure the services have rejoined the cluster
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cumin1001:~$ sudo cumin 'cloudcephmon1001.wikimedia.org' 'systemctl restart ceph-mon@cloudcephmon1001'
user@cumin1001:~$ sudo cumin 'cloudcephmon1001.wikimedia.org' 'ceph -s'
user@cumin1001:~$ sudo cumin 'cloudcephmon1002.wikimedia.org' 'systemctl restart ceph-mon@cloudcephmon1002'
user@cumin1001:~$ sudo cumin 'cloudcephmon1002.wikimedia.org' 'ceph -s'
user@cumin1001:~$ sudo cumin 'cloudcephmon1003.wikimedia.org' 'systemctl restart ceph-mon@cloudcephmon1003'
user@cumin1001:~$ sudo cumin 'cloudcephmon1003.wikimedia.org' 'ceph -s'
&lt;/syntaxhighlight&gt;

Example healthy output from ceph -s
 ...
   services:                                                                                                                                                                                                                                                        
    mon: 3 daemons, quorum cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 (age 7s)                                                                                                                                                                             
    mgr: cloudcephmon1002(active, since 2m), standbys: cloudcephmon1001, cloudcephmon1003                                                                                                                                                                          
    osd: 24 osds: 24 up (since 10w), 24 in (since 10w)     
 ...

==== ceph-osd ====
&lt;span style=&quot;background:#ff0000 &quot;&gt;IMPORTANT: This process should be run by the WMCS team&lt;/span&gt;

&lt;span style=&quot;background:#ff0000 &quot;&gt;IMPORTANT: Restarting multiple ceph-osd processes together can cause service outage or heavy network utilization during data rebalancing.&lt;/span&gt;

Until we have more Ceph OSD servers, we'll want to restart each OSD one at a time to ensure the cluster remains healthy.

Identify the systemctl unit files for each OSD
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cloudcephosd1001:~$ systemctl | grep ceph-osd@[0-9]
 ceph-osd@0.service                                                                               loaded active running   Ceph object storage daemon osd.0                                             
 ceph-osd@1.service                                                                               loaded active running   Ceph object storage daemon osd.1                                             
 ceph-osd@12.service                                                                              loaded active running   Ceph object storage daemon osd.12                                            
 ceph-osd@15.service                                                                              loaded active running   Ceph object storage daemon osd.15                                            
 ceph-osd@18.service                                                                              loaded active running   Ceph object storage daemon osd.18                                            
 ceph-osd@21.service                                                                              loaded active running   Ceph object storage daemon osd.21                                            
 ceph-osd@3.service                                                                               loaded active running   Ceph object storage daemon osd.3                                             
 ceph-osd@9.service                                                                               loaded active running   Ceph object storage daemon osd.9  
&lt;/syntaxhighlight&gt;

Restart the ceph-osd services one by one
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cloudcephosd1001:~$ sudo systemctl restart ceph-osd@0
&lt;/syntaxhighlight&gt;

Ceph health after restarting a service
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cloudcephosd1001:~$ sudo ceph -s
  cluster:
    id:     5917e6d9-06a0-4928-827a-f489384975b1
    health: HEALTH_WARN
            1 osds down
 ...
&lt;/syntaxhighlight&gt;

Once the OSD is back online the data verification begins
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cloudcephosd1001:~$ sudo ceph -s
  cluster:
    id:     5917e6d9-06a0-4928-827a-f489384975b1
    health: HEALTH_WARN
            Degraded data redundancy: 1842/123657 objects degraded (1.490%), 11 pgs degraded
 ...
&lt;/syntaxhighlight&gt;

Within a few seconds the OSD should be fully back online and healthy
&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
user@cloudcephosd1001:~$ sudo ceph -s
  cluster:
    id:     5917e6d9-06a0-4928-827a-f489384975b1
    health: HEALTH_OK
 ...
&lt;/syntaxhighlight&gt;

=== Removing or Replacing a failed OSD drive ===
The process to remove or replace a failed OSD are the same. Both the OSD configuration and service are removed from the storage cluster.

&lt;ol&gt;
&lt;li&gt;Locate the failed OSD (osd.0 in this example)
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cloudcephmon1001:~$ sudo ceph osd tree
ID CLASS WEIGHT   TYPE NAME                 STATUS REWEIGHT PRI-AFF 
-1       41.90625 root default                                      
-3       13.96875     host cloudcephosd1001                         
 0   ssd  1.74609         osd.0               down  1.00000 1.00000 
 1   ssd  1.74609         osd.1                 up  1.00000 1.00000 
 3   ssd  1.74609         osd.3                 up  1.00000 1.00000 
 9   ssd  1.74609         osd.9                 up  1.00000 1.00000 
12   ssd  1.74609         osd.12                up  1.00000 1.00000 
15   ssd  1.74609         osd.15                up  1.00000 1.00000 
18   ssd  1.74609         osd.18                up  1.00000 1.00000 
21   ssd  1.74609         osd.21                up  1.00000 1.00000 
-5       13.96875     host cloudcephosd1002                         
 2   ssd  1.74609         osd.2                 up  1.00000 1.00000 
 4   ssd  1.74609         osd.4                 up  1.00000 1.00000 
 5   ssd  1.74609         osd.5                 up  1.00000 1.00000 
10   ssd  1.74609         osd.10                up  1.00000 1.00000 
13   ssd  1.74609         osd.13                up  1.00000 1.00000 
16   ssd  1.74609         osd.16                up  1.00000 1.00000 
19   ssd  1.74609         osd.19                up  1.00000 1.00000 
22   ssd  1.74609         osd.22                up  1.00000 1.00000 
-7       13.96875     host cloudcephosd1003                         
 6   ssd  1.74609         osd.6                 up  1.00000 1.00000 
 7   ssd  1.74609         osd.7                 up  1.00000 1.00000 
 8   ssd  1.74609         osd.8                 up  1.00000 1.00000 
11   ssd  1.74609         osd.11                up  1.00000 1.00000 
14   ssd  1.74609         osd.14                up  1.00000 1.00000 
17   ssd  1.74609         osd.17                up  1.00000 1.00000 
20   ssd  1.74609         osd.20                up  1.00000 1.00000 
23   ssd  1.74609         osd.23                up  1.00000 1.00000 
&lt;/syntaxhighlight&gt;
&lt;/li&gt;
&lt;li&gt;Stop the ceph-osd service on the server with the failed OSD

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cloudcephosd1001:~$ sudo systemctl stop ceph-osd@0
&lt;/syntaxhighlight&gt;
&lt;/li&gt;
&lt;li&gt;Remove the OSD from Ceph
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cloudcephosd1001:~$ sudo ceph osd out 0
&lt;/syntaxhighlight&gt;
&lt;/li&gt;
&lt;li&gt;Verify that Ceph is rebuilding the placement groups
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cloudcephosd1001:~$ sudo ceph health -w
&lt;/syntaxhighlight&gt;
&lt;/li&gt;
&lt;li&gt;Remove the OSD from the CRUSH map
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cloudcephosd1001:~$ sudo ceph osd crush remove osd.0
&lt;/syntaxhighlight&gt;
&lt;/li&gt;
&lt;li&gt;Remove the OSD authentication keys
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cloudcephosd1001:~$ sudo ceph auth del osd.0
&lt;/syntaxhighlight&gt;
&lt;/li&gt;
&lt;li&gt;Remove the OSD from Ceph
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cloudcephosd1001:~$ sudo ceph osd rm osd.0
&lt;/syntaxhighlight&gt;
&lt;/li&gt;
&lt;li&gt;Replace the physical drive&lt;/li&gt;
&lt;li&gt;Follow the process to add a new OSD once the drive is replaced&lt;/li&gt;
&lt;/ol&gt;


=== Upgrading mon nodes ===

==== OS upgrade ====

==== Ceph version upgrade ====

Once all ceph mons are running the same OS, and the next ceph version is available on that OS, upgrading ceph packages can be done on all nodes in sequence using the wmcs.ceph.upgrade_mons cookbook. This will upgrade all nodes to whatever ceph version is specified in the hiera setting '''profile::cloudceph::ceph_repository_component'''

Ceph runs version integrity checks which will produce a health alert if some nodes are running a newer ceph version than expected. You can change ceph's expectations about version by running this on a mon node:

 # ceph osd require-osd-release &lt;next targeted version&gt;

Despite having 'require' in the name, this will actually only complain if any nodes are running something newer than the targeted version.


As long as there are three healthy mon nodes, a mon node can be upgraded using a standard re-image process without any advance or followup work. Nodes must be upgraded one at a time, and ceph must be in a fully healthy state between upgrades.

=== Replacing MON nodes ===

An additional MON node should be added to the cluster before removing the node to be refreshed. There are several places in Hiera that enumerate mons; these must be updated to in include the new node. See [https://gerrit.wikimedia.org/r/c/operations/puppet/+/1268280 an example of doing that].

=== Upgrading OSD nodes ===

==== Ceph version upgrade ====

Once all ceph mons are running the same OS, and the next ceph version is available on that OS, upgrading ceph packages can be done on all nodes in sequence using the wmcs.ceph.upgrade_osds cookbook.

==== OS upgrade ====

Most OSD nodes use the 'partman/custom/cephosd.cfg' partman recipe which replaces OS drives while leaving OSD partitions intact. Typically it's possible for a re-imaged OSD node to rejoin the cluster after reimaging with a minimum of rebalancing. To replace a reimaged node in the cluster, use the 'wmcs.ceph.osd.reactivate' cookbook; this re-initializes ceph settings on the node that were stored on the OS partition and destroyed during reimage.

A full reimage process generally looks like this:

* nic firmware upgrade from a production cumin node:
 $ sudo cookbook sre.hardware.upgrade-firmware  &quot;cloudcephosd1006.eqiad.wmnet&quot; --new -c nic
* wiping of OS drives in the bios raid controller config (typically without this the partman recipe fails during re-image)
* full re-image on a production cumin node
 sudo cookbook sre.hosts.reimage --os bookworm cloudcephosd1006
* re-activation (on a cloudcumin node) using wmcs.ceph.osd.reactivate
 sudo cookbook wmcs.ceph.osd.reactivate --cluster-name eqiad1 --osd-hostname cloudcephosd1006 --skip-reboot

== Clients ==

=== Rate limiting ===
==== Native ====
Native RBD rate limiting is supported in the Ceph Nautilus release. Due to upstream availability and multiple Debian releases we will likely have a mixture of older Ceph client versions during phase1.

Available rate limiting options and their defaults in the Nautilus release:

 $ rbd config pool ls &lt;pool&gt; | grep qos
 rbd_qos_bps_burst                                   0         config
 rbd_qos_bps_limit                                   0         config
 rbd_qos_iops_burst                                  0         config
 rbd_qos_iops_limit                                  0         config
 rbd_qos_read_bps_burst                              0         config
 rbd_qos_read_bps_limit                              0         config
 rbd_qos_read_iops_burst                             0         config
 rbd_qos_read_iops_limit                             0         config
 rbd_qos_schedule_tick_min                           50        config
 rbd_qos_write_bps_burst                             0         config
 rbd_qos_write_bps_limit                             0         config
 rbd_qos_write_iops_burst                            0         config
 rbd_qos_write_iops_limit                            0         config

==== OpenStack ====
IO rate limiting can also be managed using a flavor's metadata. This will trigger libvirt to apply `iotune` limits on the ephemeral disk.

;Available disk tuning options:
* disk_read_bytes_sec
* disk_read_iops_sec
* disk_write_bytes_sec
* disk_write_iops_sec
* disk_total_bytes_sec
* disk_total_iops_sec

Example commands to create or modify flavors metadata with rate limiting options roughly equal to a 7200RPM SATA Disk:

 openstack flavor create \
   --ram 2048 \
   --disk 20 \
   --vcpus 1 \
   --private \
   --project testlabs \
   --id 857921a5-f0af-4069-8ad1-8f5ea86c8ba2 \
   --property quota:disk_total_iops_sec=200 m1.small-ceph

 openstack flavor set --property quota:disk_total_bytes_sec=250000000 857921a5-f0af-4069-8ad1-8f5ea86c8ba2

The Ceph host aggregate will pin flavors to the Hypervisors configured to use Ceph

 openstack flavor set --property aggregate_instance_extra_specs:ceph=true 857921a5-f0af-4069-8ad1-8f5ea86c8ba2

Example rate limit configuration as seen by libvirt. (virsh dumpxml &lt;instance name&gt;)
 &lt;target dev='sda' bus='virtio'/&gt;
  &lt;iotune&gt;
   &lt;total_bytes_sec&gt;250000000&lt;/total_bytes_sec&gt;
   &lt;total_iops_sec&gt;200&lt;/total_iops_sec&gt;
  &lt;/iotune&gt;

'''''NOTE''': Updating a flavors metadata does not have any effect on existing virtual machines.''

When a virtual machine is created the requested flavor data is copied to the instance, and any future updates to the flavor are ignored. This script will connect to the Nova database and directly modify extra specs.
 $ wmcs-vm-extra-specs --help
 usage: wmcs-vm-extra-specs [-h] [--nova-db-server NOVA_DB_SERVER] [--nova-db NOVA_DB] [--mysql-password MYSQL_PASSWORD]
                           uuid
                           {quota:disk_read_bytes_sec,
                            quota:disk_read_iops_sec,
                            quota:disk_total_bytes_sec,
                            quota:disk_total_iops_sec,
                            quota:disk_write_bytes_sec,
                            quota:disk_write_iops_sec}
                           spec_value

=== Glance ===

Glance images used by Ceph based virtual machines should have the hw_scsi_model=virtio-scsi hw_disk_bus=scsi properties defined. These properties will enable SCSI discard operations and instruct Ceph to remove blocks that have been deleted. 

Image create example
 openstack image create --file debian-buster-20191218.qcow2 \
   --disk-format &quot;qcow2&quot; \
   --property hw_scsi_model=virtio-scsi \
   --property hw_disk_bus=scsi \
   --public \
   debian-10.0-buster

(Once Glance is configured to store images in Ceph, the disk-format will change from qcow2 to raw)

=== Nova compute ===

==== Migrating local VMs to Ceph ====

Switch puppet roles to the Ceph enabled wmcs::openstack::eqiad1::virt_ceph role.
In operations/puppet/manifest/site.pp:
 node 'cloudvirt1022.eqiad.wmnet' {
    role(wmcs::openstack::eqiad1::virt_ceph)
 }

Run the puppet agent on the hypervisor
 hypervisor $ sudo puppet agent -tv

Shutdown the VM
 cloudcontrol $ openstack server stop &lt;UUID&gt;

Convert the local QCOW2 image to raw and upload to Ceph
 hypervisor $ qemu-img convert -f qcow2 -O raw /var/lib/nova/instances/&lt;UUID&gt;/disk rbd:compute/&lt;UUID&gt;_disk:id=eqiad1-compute

Undefine the virtual machine. This command removes the existing libvirt definition from the hypervisor, once nova attempts to start the VM it will be redefined with the RBD configuration. (This step can be ignored, but you may notice some errors in nova-compute.log until the VM has been restarted)
 hypervisor $ virsh undefine &lt;OS-EXT-SRV-ATTR:instance_name&gt;

Cleanup local storage files
 hypervisor $ rm /var/lib/nova/instances/&lt;UUID&gt;/disk
 hypervisor $ rm /var/lib/nova/instances/&lt;UUID&gt;/disk.info

Power on the VM
 cloudcontrol $ openstack server start &lt;UUID&gt;

----

'''Reverting back to local storage''' 

Shutdown the VM
 cloudcontrol $ openstack server stop &lt;UUID&gt;

Convert the Ceph raw image back to QCOW2 on the local hypervisor
 hypervisor $ qemu-img convert -f raw -O qcow2 rbd:compute/&lt;UUID&gt;_disk:id=eqiad1-compute /var/lib/nova/instances/UUID/disk

Power on the VM
 cloudcontrol $ openstack server start &lt;UUID&gt;

==== CPU Model Type ====
A virtual machine can only be live migrated to a hypervisor matching the same CPU. CloudVPS currently has multiple CPU models and is using the default &quot;host-model&quot; nova configuration.

To enable live migration between any production hypervisor, the cpu_mode parameter should match the lowest hypervisor CPU model.

{| class=&quot;wikitable&quot;
!style=&quot;text-align:left;&quot;|Hypervisor range
!CPU model
!Launch date
|-
|cloudvirt[1023-1030].eqiad.wmnet
|Gold 6140 Skylake
|2017
|-
|cloudvirt[1016-1022].eqiad.wmnet
|E5-2697 v4 Broadwell
|2016
|-
|cloudvirt[1012-1014].eqiad.wmnet
|E5-2697 v3 Haswell
|2014
|-
|cloudvirt[1001-1009].eqiad.wmnet
|E5-2697 v2 Ivy Bridge
|2013
|}

==== Virtual Machine Images ====

Important: Using QCOW2 for hosting a virtual machine disk is NOT recommended. If you want to boot virtual machines in Ceph (ephemeral backend or boot from volume), please use the raw image format within Glance.

Once all CloudVPS virtual machines have been migrated to Ceph we can convert the existing virtual machine images in Glance from QCOW2 to raw. This will avoid having nova-compute convert the image each time a new virtual machine is created.

==== Object Storage via Radosgw ====

Cloud-vps provides S3 and swift-compatible [[object storage]] with auth and discovery managed by [[Help:Using OpenStack APIs|Openstack Keystone]]. These APIs are served by the [https://docs.ceph.com/en/quincy/radosgw/ Ceph rados gateway].

Radosgw runs on cloudcontrols and is configured using standard ceph config files. Its keyring is /etc/ceph/ceph.client.radosgw.keyring and its major config is in the &lt;nowiki&gt;[client.radosgw]&lt;/nowiki&gt; section of /etc/ceph/ceph.conf. Both of the above are managed by puppet.

Even though the swift object storage endpoint emulates and openstack service (and can be discovered via keystone), it is managed by different commands. For example, [[Portal:Cloud VPS/Admin/Projects lifecycle#swift_/_S3_/_radosgw_/_object_storage|radosgw quotas are managed with the radosgw-admin tool]].

==== VirtIO SCSI devices ====

Currently CloudVPS virtual machines are configured with the virtio-blk driver. This driver does not support discard/trim operations to free up deleted blocks.

Discard support can be enabled by using the virtio-scsi driver, but it's important to note that the device labels will change from /dev/vda to /dev/sda.

==== IO Throttles in Nova Flavors  ====

To avoid bandwidth and iops contention between VMs, most VMs are throttled with the following limits:

disk_total_bytes_sec: 10000000
disk_read_iops_sec: 5000
disk_write_iops_sec: 500

Any newly created flavor should be altered with the following commands:

 openstack flavor set --property aggregate_instance_extra_specs:ceph=true &lt;flavorid&gt;
 openstack flavor set --property quota:disk_total_bytes_sec='200000000' &lt;flavorid&gt;
 openstack flavor set --property quota:disk_write_iops_sec='500' &lt;flavorid&gt;
 openstack flavor set --property quota:disk_read_iops_sec='5000' &lt;flavorid&gt;

Those numbers are based on the following assumptions:

* Support up to 1000 VMs of which up to third may be reading or writing at any one instant
* OSD drives support 42000 write iops and 74000 read iops
** source: https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/dell-poweredge-sas-ssd-performance-specifications.pdf
* Total ceph cluster supports up to 560000 write iops and 8880000 read iops
** 42000 * 120 (OSDs) /3 /3 = 560000
** 74000 * 120 = 8880000
** source for these formulas: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015479.html
* 10Gb/s duplex networking divided among all VMs
** The 'total_bytes_sec' metric assumes many fewer VMs are reading or writing flat out -- 1/20th the number used for iops.  This is based partly on optimism and partly on new instance creation being unacceptably slow with a smaller number.

Some of the math for all this can be found at https://docs.google.com/spreadsheets/d/1_fRqCLLA8zBJP9pFnPEj4x-6aG5yM_ZMFbTHPZTPSeI/edit#gid=1136514968

== Performance Testing ==
=== Network ===
Jumbo frames (9k MTU) have been configured to improve the network throughput and overall network performance, as well as reduce the CPU utilization on the Ceph OSD servers. 

==== Baseline (default tuning options) ====

Iperf options used to simulate Ceph storage IO.
 -N disable Nagle's Algorithm
 -l 4M set read/write buffer size to 4 megabyte
 -P number of parallel client threads to run (one per OSD)

Server:
 iperf -s -N -l 4M

Client: 
 iperf -c &lt;server&gt; -N -l 4M -P 8

===== cloudcephosd &lt;-&gt; cloudcephosd =====
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec  2.74 GBytes  2.35 Gbits/sec
 [ 10]  0.0-10.0 sec  2.74 GBytes  2.35 Gbits/sec
 [  9]  0.0-10.0 sec   664 MBytes   557 Mbits/sec
 [  6]  0.0-10.0 sec   720 MBytes   603 Mbits/sec
 [  5]  0.0-10.0 sec  1.38 GBytes  1.18 Gbits/sec
 [ 13]  0.0-10.0 sec  1.38 GBytes  1.18 Gbits/sec
 [  7]  0.0-10.0 sec   720 MBytes   602 Mbits/sec
 [  8]  0.0-10.0 sec   720 MBytes   603 Mbits/sec
 [SUM]  0.0-10.0 sec  11.0 GBytes  9.42 Gbits/sec

===== cloudvirt1022 -&gt; cloudcephosd =====
 cloudvirt1022 &lt;-&gt; cloudcephosd: 8.55 Gbits/sec
 [ ID] Interval       Transfer     Bandwidth
 [  7]  0.0-10.0 sec  1.11 GBytes   949 Mbits/sec
 [  6]  0.0-10.0 sec  1.25 GBytes  1.07 Gbits/sec
 [  4]  0.0-10.0 sec  1.39 GBytes  1.19 Gbits/sec
 [  9]  0.0-10.0 sec  1.24 GBytes  1.06 Gbits/sec
 [ 10]  0.0-10.0 sec  1.07 GBytes   920 Mbits/sec
 [  5]  0.0-10.0 sec  1.36 GBytes  1.16 Gbits/sec
 [  3]  0.0-10.0 sec  1.41 GBytes  1.21 Gbits/sec
 [  8]  0.0-10.0 sec  1.17 GBytes  1.00 Gbits/sec
 [SUM]  0.0-10.0 sec  10.0 GBytes  8.55 Gbits/sec

=== Ceph RBD ===
To browse an exhaustive set of performance tests you can go to the [https://cloud-ceph-performance-tests.toolforge.org/ cloud ceph test result explorer].

To run the tests you can [https://gerrit.wikimedia.org/r/admin/repos/cloud/storage-performance-tests download the scripts and code].

== CLI examples ==

Create, format and mount a RBD image (useful for testing / debugging)
 $ rbd create datatest --size 250 --pool compute --image-feature layering
 $ rbd map datatest --pool compute --name client.admin
 $ mkfs.ext4 -m0 /dev/rbd0
 $ mount /dev/rbd0 /mnt/

 $ umount /mnt
 $ rbd unmap /dev/rbd0
 $ rbd rm compute/datatest

-----

List RBD nova images
 $ rbd ls -p compute
 9e2522ca-fd5e-4d42-b403-57afda7584c0_disk

-----

Show RBD image information 
 $ rbd info -p compute 9051203e-b858-4ec9-acfd-44b9e5c0ecb1_disk
 rbd image '9051203e-b858-4ec9-acfd-44b9e5c0ecb1_disk':
        size 20 GiB in 5120 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: aec56b8b4567
        block_name_prefix: rbd_data.aec56b8b4567
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features:
        flags:
        create_timestamp: Mon Jan  6 21:36:11 2020
        access_timestamp: Mon Jan  6 21:36:11 2020
        modify_timestamp: Mon Jan  6 21:36:11 2020

-----

View RBD image with qemu tools on a hypervisor
 $ qemu-img info rbd:&lt;pool&gt;/&lt;vm uuid&gt;_disk:id=&lt;ceph user&gt;</text>
      <sha1>ab3nj0289rmn2y6e47zrptcn529gq59</sha1>
    </revision>
  </page>
  <page>
    <title>Gerrit/Operations</title>
    <ns>0</ns>
    <id>447761</id>
    <revision>
      <id>2414324</id>
      <parentid>2403452</parentid>
      <timestamp>2026-05-16T22:09:40Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414324</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="16798" sha1="2j3ioda0fktz5a0gp4jcv3qak3ff75e" xml:space="preserve">{{gerrit nav}}

== Restarting ==

Restarting Gerrit is a last resort. We used to have to restart it often due to misunderstanding of some of its behavior as well as [https://phabricator.wikimedia.org/phame/post/view/232/tracking_memory_issue_in_a_java_application/a nasty memory leak]. As of February 2021, restart should not be conducted without a thorough review of the current behavior and taking traces. They will be of dramatic help to identify a potential bug or a configuration tuning.

If after all investigations you get clueless or really have no other options, you can restart Gerrit through a cookbook run: &lt;code&gt;sudo cookbook sre.gerrit.restart-gerrit --host gerrit2002&lt;/code&gt;.

The service will take a few seconds before it comes back during which any end user operations would error out (some Puppet catalogues, CI, developers).

== Monitoring ==

JavaMelody monitors the state of the Gerrit JVM. They are collected by Prometheus from https://gerrit.wikimedia.org/r/monitoring?prometheus

* [[gerrit:monitoring|JavaMelody in Gerrit]] (only accesible to logged-in Gerrit Administrators/Gerrit Managers)
* [https://grafana.wikimedia.org/dashboards/f/5AnaHr2Mk Grafana &quot;Gerrit&quot; folder&quot;]
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?orgId=1 Grafana Gerrit JavaMelody Dashboard]
** [https://grafana.wikimedia.org/d/uXZMn9PWz Overview]

=== Important Graphs ===

* [https://grafana.wikimedia.org/goto/UX_fk03Ig?orgId=1 Gerrit overview dashboard]
* Memory Usage
** [https://gerrit.wikimedia.org/r/monitoring?part=graph&amp;graph=usedMemory Gerrit Used Memory]
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=12&amp;fullscreen&amp;orgId=1&amp;var-Application=&amp;var-Window=5m Grafana Used Memory]
* GC Timing
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=14&amp;fullscreen&amp;orgId=1 Grafana GC Timing]
** Garbage collection metrics. Times in the 100s of milliseconds, rather than in the 10s of milliseconds can be indicative of a problem (running low on memory)
* Active Threads
** [https://gerrit.wikimedia.org/r/monitoring?part=graph&amp;graph=activeThreads Gerrit Active Threads]
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=16&amp;fullscreen&amp;orgId=1&amp;var-Application=&amp;var-Window=5m Grafana Active Threads]
** Usually there are less than 20 active threads at any given time — more than that typically means that you should take a [[#Thread_Dump|Thread Dump]] and restart.

== Gerrit metrics ==

On top of the JavaMelody data, [[gerrit:Documentation/metrics.html|Gerrit has internal metrics]].

For users having the [[gerrit:Documentation/access-control.html#capability_viewCaches|| viewCaches]] or [[gerrit:plugins/metrics-reporter-prometheus/Documentation/config.md|View Metrics]] capabilities, various internal Gerrit metrics can be retrieved via:

* https://gerrit.wikimedia.org/r/config/server/metrics

Which obviously requires authentication. That complements &lt;code&gt;gerrit show-caches&lt;/code&gt;.

We use the metrics-reporter-prometheus plugin which exposes  collected by Prometheus from https://gerrit.wikimedia.org/r/plugins/metrics-reporter-prometheus/metrics . Those Gerrit metrics can also be seen on [https://gerrit.wikimedia.org/r/monitoring?part=mbeans the JavaMelody MBeans page] under the &lt;code&gt;metrics&lt;/code&gt; branch.

See [https://grafana.wikimedia.org/dashboards/f/5AnaHr2Mk Gerrit Grafana dashboards folder].

== Use &lt;code&gt;gerrit ssh&lt;/code&gt; commands ==
To use Gerrit ssh commands (see [https://gerrit-review.googlesource.com/Documentation/cmd-index.html#user_commands command list documentation]), you will need to be a member of the ldap group gerritadmin. To do so, (more info on the [[SRE/LDAP]] documentation), you will need to run &lt;code&gt;ldapvi -b ou=groups cn=gerritadmin&lt;/code&gt; and then wait for the synchronization to happen. Then, you'll be able to run commands such as [[Gerrit/Operations#Killing ssh connections]] or &lt;code&gt;ssh user@gerrit.wikimedia.org -p 29418 gerrit show-caches&lt;/code&gt; etc.
{{Warn
| content = Please avoid running replication command without asking RelEng or Collab first (see https://wikitech.wikimedia.org/wiki/Gerrit/Administration#Forcing_Replication_re-runs)
}}

== Logs ==

They are consumed by our logging infrastructure and available in the [https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV Kibana dashboard for Gerrit (application logs)] and [https://logstash.wikimedia.org/goto/e79e3841a2cd0cc17ed16481bd5fb1da Apache access logs].

=== Main logs ===
Logs are available on the gerrit servers at: &lt;code&gt;/var/log/gerrit/&lt;/code&gt;. There are a number of logfiles:

* &lt;code&gt;gerrit.log&lt;/code&gt;: This is the main log file and will show stacktraces and errors
* &lt;code&gt;gerrit.json&lt;/code&gt;: Like &lt;code&gt;gerrit.log&lt;/code&gt; bug not really human readable. For sending structured logs to logstash.
* &lt;code&gt;sshd_log&lt;/code&gt;: Log of sshd events
* &lt;code&gt;gc_log&lt;/code&gt;: Logs for &lt;code&gt;git gc&lt;/code&gt; '''not''' the JVM garbage collection (those logs are available in &lt;code&gt;/srv/gerrit/jvmlogs&lt;/code&gt;)
* &lt;code&gt;plugin_log&lt;/code&gt;: Info about plugins being loaded and reloaded, this information is also in &lt;code&gt;gerrit.log&lt;/code&gt;

=== HTTP Logs ===

Gerrit sits behind Apache, access and error logs are both in &lt;code&gt;/var/log/apache2&lt;/code&gt;:

* &lt;code&gt;gerrit.wikimedia.org.https.access.log&lt;/code&gt;
* &lt;code&gt;gerrit.wikimedia.org.https.error.log&lt;/code&gt;
find its logs by searching with &lt;code&gt;type:log4j&lt;/code&gt;.

== JVM ==

=== Thread Dump ===

A thread dump is often useful in troubleshooting. To capture a thread dump use &lt;code&gt;jstack&lt;/code&gt;. This code should be safe to run at any time, and is run frequently while Gerrit is running:

 sudo -u gerrit2 jstack -l $(pgrep java) &gt; &quot;/srv/gerrit/jstack-$(date +%Y-%m-%d-%H%M%S).dump&quot;

It's often useful to upload the resulting file to https://fastthread.io/ to detect problems.

=== Java trace ===
{{warn|This command isn't run very often, unsure how safe it is to run; kept here for folks who are familiar with jstat}}

Display a summary of garbage collection statistics every 1000 ms:

 sudo -u gerrit2 /usr/lib/jvm/java-8-openjdk-amd64/bin/jstat -gcutil &quot;$(pgrep -u gerrit2 java)&quot; 1000

=== Java heap usage ===

''Requires openjdk-X-dbg for the debugging symbols''

   sudo /usr/lib/jvm/java-8-openjdk-amd64/bin/jmap -heap &quot;$( pgrep -u gerrit2 java)&quot;

== Access h2 account_patch_reviews ==

On copies of account_patch_reviews* files:
&lt;pre&gt;
java -cp h2-1.3.176.jar org.h2.tools.Shell -url jdbc:h2:/home/hashar/account_patch_reviews
&lt;/pre&gt;
Which gives you a sql prompt:
&lt;pre&gt;
sql&gt; show columns from ACCOUNT_PATCH_REVIEWS
...&gt; ;
FIELD        | TYPE         | NULL | KEY | DEFAULT
ACCOUNT_ID   | INTEGER(10)  | NO   | PRI | 0
CHANGE_ID    | INTEGER(10)  | NO   | PRI | 0
PATCH_SET_ID | INTEGER(10)  | NO   | PRI | 0
FILE_NAME    | VARCHAR(255) | NO   | PRI | ''
(4 rows, 16 ms)
&lt;/pre&gt;

== Blocking misbehaving bots / IPs ==

If necessary either IP addresses or user agents that are misbehaving can be blocked by making edits to '''modules/profile/templates/gerrit/apache.erb''' in the '''operations/puppet''' public git repository
and merging them.

[https://gerrit.wikimedia.org/r/c/operations/puppet/+/1037856/5/modules/profile/templates/gerrit/apache.erb example change]

=== Throttling IPs ===

Since September 2024, implemented in [[phab:T365259]] there is another method of throttling abusive traffice using [[nftables]]. See [[Firewall#Throttling_with_nftables]] and the [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/gerrit.yaml profile::firewall::nftables_throttling keys in Hiera].

You can also observe data related to this on the [https://grafana.wikimedia.org/goto/iU0RPFiHR?orgId=1 grafana dashboard for gerrit].

== Killing ssh connections ==

It can happen that a user reaches the limit of 8 concurrent ssh connections and then says they can't push to Gerrit anymore over ssh.

A member of Gerrit admins can run commands like these to kill connections for them:


 ssh user@gerrit.wikimedia.org -p 29418 gerrit show-connections
 ssh user@gerrit.wikimedia.org -p 29418 gerrit close-connection &lt;connection ID&gt;

== Failover (in case primary host is unavailable) ==

# Disable puppet on the elected replacement
# Enable read only with the cookbook&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
sudo cookbook sre.gerrit.read-only-toggle --host gerrit1003 --toggle on
&lt;/syntaxhighlight&gt;
# Run the local_backup cookbook ([[Gerrit/Operations#Warming up local backup|Gerrit/Operations#Warming_up_local_backup)]]&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
sudo cookbook sre.gerrit.localbackup --source gerrit1003
&lt;/syntaxhighlight&gt;
# Switch role locally on the gerrit systemd unit with &lt;syntaxhighlight lang=&quot;shell&quot;&gt;
systemctl edit gerrit.service --full
&lt;/syntaxhighlight&gt;
## Remove &lt;code&gt;--replica&lt;/code&gt; from &lt;code&gt;ExecStart=&lt;/code&gt;
## Remove &lt;code&gt;--enable-httpd&lt;/code&gt; from &lt;code&gt;ExecStart=&lt;/code&gt;
## Save and quit
## Run &lt;code&gt;systemctl daemon-reload&lt;/code&gt;
# Switch the &lt;code&gt;CNAME&lt;/code&gt; discovery record in our DNS repo in [[gerrit:plugins/gitiles/operations/dns/+/refs/heads/master/templates/wmnet#1044|templates/wmnet#1044]] as described in [[phab:T387833#11601246|this task]] to designate the newly promoted replacement. For this you'll need to follow [[DNS#Emergency Measures]] 
# Restart gerrit with &lt;code&gt;systemctl restart gerrit &amp;&amp; journalctl -fln50 -u gerrit&lt;/code&gt;
# Confirm that you can now access the [https://gerrit.wikimedia.org new primary] and clone projects both via SSH and HTTPS&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
git clone --depth=1 https://gerrit.wikimedia.org/r/mediawiki/extensions.git
git clone --depth=1 &quot;ssh://gerrit.wikimedia.org:29418/mediawiki/extensions&quot;

&lt;/syntaxhighlight&gt;
# Disable read only mode:&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
sudo cookbook sre.gerrit.read-only-toggle --host gerrit1003 --toggle off
&lt;/syntaxhighlight&gt;
# Backport the changes to puppet
# Resume replication

== Switch replicas over ==


In [[phab:T406334|T406334]] we switched replicas, editing DNS and Puppet as follows:

* [[gerrit:c/operations/dns/+/1242268|operations/dns/+/1242268]] (and its hotfix [[gerrit:c/1243119|1243119]])
* [[gerrit:c/operations/puppet/+/1242269|operations/puppet/+/1242269]] 


== Switchover (planned maintenance) ==
{{Infobox2|data1=If you're migrating on gerrit2003, please see https://phabricator.wikimedia.org/T338470#10506291 
Try to ensure the user running Gerrit will be also owning the synced data.}}
=== Schedule and Announce Downtime ===
Announce the scheduled downtime for Gerrit services. 

{{Warning
| content = Please also take note of that specific section of Gerrit's administration page: https://wikitech.wikimedia.org/wiki/Gerrit/Administration#Forcing_Replication_re-runs
}}

=== Prepare patches ===
Those hiera keys have to be updated:
 profile::gerrit::active_host
 profile::gerrit::replica_hosts
 profile::gerrit::lfs_sync_dest
(see [[gerrit:c/operations/puppet/+/1217133|example here]]){{Warn
| content = You can submit those patches but do not merge them.
}}
You also have to update the [[DNS|DNS configuration]] (see [[gerrit:c/operations/dns/+/1238708|example here]]).

=== Automated switchover ===
{{Caution|Be aware that warming up the transfers will erase all data on the target instance.}}

==== Warming up transfer ====
 sudo cookbook sre.gerrit.sync-instances --source gerrit1003 --replica gerrit2003 --chown --distrust
The args &lt;code&gt;--distrust&lt;/code&gt; and &lt;code&gt;--chown&lt;/code&gt; are given to bypass Gerrit internal replication, this will change in the future. Those arguments imply we run the &lt;code&gt;rsync&lt;/code&gt; commands listed below in the manual switchover procedure via the cookbook and then ensure ownership of the transferred files. 

==== Warming up local backup ====
 sudo cookbook sre.gerrit.localbackup --source gerrit1003
Here, we run a local backup to ensure having a local (on the source instance) snapshot of our data before doing anything.

==== Switching Over ====
 sudo cookbook sre.gerrit.switchover --switch-from-host gerrit1003 --switch-to-host gerrit2003 --distrust --chown
The &lt;code&gt;--distrust&lt;/code&gt; and &lt;code&gt;--chown&lt;/code&gt; args are also required here to be passed on to the other cookbooks that will be used to perform the switchover. The cookbook will guide you through the next steps, asking you to merge the Puppet patch and then the DNS patch.

=== Manual switchover ===

==== Prepare emergency rollback commands for DNS ====
Preparing the commands listed in [[DNS#Update DNS if Gerrit is down]] could help revert if needed.

==== Stop Puppet across all gerrit instances ====
 sudo disable-puppet 'gerrit maintenance'

==== Merge the puppet changes to prepare the switch over to &lt;code&gt;dst-gerrit&lt;/code&gt; ====
Merge the Puppet changes to prepare for the desired state.

==== Begin Scheduled Downtime ====
Announce the start of the scheduled downtime on IRC &lt;code&gt;#wikimedia-operations&lt;/code&gt; and on Slack &lt;code&gt;#engineering-all&lt;/code&gt;.

==== Downtime Management ====
 sudo cookbook sre.hosts.downtime -r 'maintenance' -D 30 src-gerrit.wikimedia.org &amp;&amp; sudo cookbook sre.hosts.downtime -r 'maintenance' -H 1 dst-gerrit.wikimedia.org
Manually schedule downtime for checks connected to the virtual server &quot;&lt;code&gt;gerrit.wikimedia.org&lt;/code&gt;&quot; on &lt;code&gt;icinga.wikimedia.org&lt;/code&gt;.

==== Update DNS ====
[[DNS#Deploying DNS changes|Run]] &lt;code&gt;sudo -i authdns-update&lt;/code&gt; on &lt;code&gt;ns0.wikimedia.org&lt;/code&gt;, review the diff '''but do not commit yet'''.

==== Stop Gerrit on &lt;code&gt;dst-gerrit&lt;/code&gt; ====
Execute the following commands on &lt;code&gt;dst-gerrit&lt;/code&gt;:
 sudo systemctl stop gerrit

==== Make source and destination read-only ====
Execute the following commands on '''both instances''':
 touch /etc/gerrit/gerrit.readonly
That will disable write operation and offer some safety from data corruption

==== Data Synchronization from &lt;code&gt;src-gerrit&lt;/code&gt; ====
 rsync -avpPz --delete /var/lib/gerrit2/review_site/ rsync://dst-gerrit.wikimedia.org/gerrit-var-lib/
 rsync -avpPz --delete /srv/gerrit/ rsync://dst-gerrit.wikimedia.org/gerrit-data/ --exclude=*.hprof

==== Stop Gerrit on &lt;code&gt;src-gerrit&lt;/code&gt; ====
Execute the following commands on &lt;code&gt;src-gerrit&lt;/code&gt;:
 sudo systemctl stop gerrit

==== Repeat Data Synchronization on &lt;code&gt;src-gerrit&lt;/code&gt; ====
Repeat the rsync commands as in this [[Gerrit/Operations#Data Synchronization on src-gerrit|step]].

This is to ensure consistency across nodes.

==== Start Gerrit on &lt;code&gt;dst-gerrit&lt;/code&gt; ====
Execute the following command on &lt;code&gt;dst-gerrit&lt;/code&gt;:
 sudo systemctl start gerrit

==== Finalize DNS Update ====
Confirm the DNS change and merge it.

==== Finalize Puppet Update ====
You can also speed-up the subsequent deployment by running &lt;code&gt;sudo run-puppet-agent&lt;/code&gt; on &lt;code&gt;dst-gerrit&lt;/code&gt;. 

=== Testing ===
* HTTPS access via browser: https://gerrit.wikimedia.org
* HTTPS access via browser: https://gerrit-replica.wikimedia.org
* SSH access: &lt;code&gt;ssh '''''yourusernameongerrit'''''@gerrit.wikimedia.org -p 29418&lt;/code&gt;

=== Announce Downtime Conclusion ===
Announce that the downtime is over.

=== Post-Migration Tasks ===

# Check if replication is running: &lt;code&gt;ssh gerrit.wikimedia.org -p 29418 gerrit show-queue --by-queue --wide&lt;/code&gt;
# Validate that https://gerrit-replica.wikimedia.org/ returns a HTTP/404 error on &lt;code&gt;/&lt;/code&gt;
# Determine the grace period duration.
# Ensure &lt;code&gt;src-gerrit&lt;/code&gt; has Puppet disabled and/or services are masked, if it needs to be decommissioned.
# If needed, decommission the old host as per [[phab:T336427|T336427]].

== Runbooks ==

=== GerritHAProxyBackendUnavailable ===
Gerrit service is not available to the tcp-proxy hosts. Most likely the active Gerrit server is down and the tcp-proxy hosts can not reach the active Gerrit server anymore. Maintenance on the active Gerrit server could be the issue and a downtime is missing for the &lt;code&gt;tcp-proxy*&lt;/code&gt; hosts.

If there is no planned maintenance the Gerrit active server should be troubleshooted (see steps above), make sure the &lt;code&gt;gerrit&lt;/code&gt; and &lt;code&gt;apache2&lt;/code&gt; and &lt;code&gt;envoy&lt;/code&gt; services are running. Also network connectivity from the tcp-proxy hosts should be possible.

=== GerritHAProxyServiceUnavailable ===
Gerrit tcp-proxy service is not available in at least one PoP and there is less than one available tcp-proxy per DC. Most likely the active Gerrit server is down and the tcp-proxy hosts can not reach the active Gerrit server anymore so they are marked as &lt;code&gt;DOWN&lt;/code&gt;. Also a PoP might be depooled but not properly downtimed.

If there is no planned maintenance the network connectivity from the tcp-proxy hosts to the Gerrit active server should be troubleshooted and make sure the &lt;code&gt;gerrit&lt;/code&gt; and &lt;code&gt;apache2&lt;/code&gt; and &lt;code&gt;envoy&lt;/code&gt; services are running.</text>
      <sha1>2j3ioda0fktz5a0gp4jcv3qak3ff75e</sha1>
    </revision>
    <revision>
      <id>2414330</id>
      <parentid>2414324</parentid>
      <timestamp>2026-05-16T22:26:47Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414330</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="16806" sha1="cy9neoje2rde8u7f8gckwnzzz2dy2n9" xml:space="preserve">{{gerrit nav}}

== Restarting ==

Restarting Gerrit is a last resort. We used to have to restart it often due to misunderstanding of some of its behavior as well as [https://phabricator.wikimedia.org/phame/post/view/232/tracking_memory_issue_in_a_java_application/a nasty memory leak]. As of February 2021, restart should not be conducted without a thorough review of the current behavior and taking traces. They will be of dramatic help to identify a potential bug or a configuration tuning.

If after all investigations you get clueless or really have no other options, you can restart Gerrit through a cookbook run: &lt;code&gt;sudo cookbook sre.gerrit.restart-gerrit --host gerrit2002&lt;/code&gt;.

The service will take a few seconds before it comes back during which any end user operations would error out (some Puppet catalogues, CI, developers).

== Monitoring ==

JavaMelody monitors the state of the Gerrit JVM. They are collected by Prometheus from https://gerrit.wikimedia.org/r/monitoring?prometheus

* [[gerrit:monitoring|JavaMelody in Gerrit]] (only accesible to logged-in Gerrit Administrators/Gerrit Managers)
* [https://grafana.wikimedia.org/dashboards/f/5AnaHr2Mk Grafana &quot;Gerrit&quot; folder&quot;]
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?orgId=1 Grafana Gerrit JavaMelody Dashboard]
** [https://grafana.wikimedia.org/d/uXZMn9PWz Overview]

=== Important Graphs ===

* [https://grafana.wikimedia.org/goto/UX_fk03Ig?orgId=1 Gerrit overview dashboard]
* Memory Usage
** [https://gerrit.wikimedia.org/r/monitoring?part=graph&amp;graph=usedMemory Gerrit Used Memory]
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=12&amp;fullscreen&amp;orgId=1&amp;var-Application=&amp;var-Window=5m Grafana Used Memory]
* GC Timing
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=14&amp;fullscreen&amp;orgId=1 Grafana GC Timing]
** Garbage collection metrics. Times in the 100s of milliseconds, rather than in the 10s of milliseconds can be indicative of a problem (running low on memory)
* Active Threads
** [https://gerrit.wikimedia.org/r/monitoring?part=graph&amp;graph=activeThreads Gerrit Active Threads]
** [https://grafana.wikimedia.org/d/Bw2mQ3iWz/gerrit-javamelody?panelId=16&amp;fullscreen&amp;orgId=1&amp;var-Application=&amp;var-Window=5m Grafana Active Threads]
** Usually there are less than 20 active threads at any given time — more than that typically means that you should take a [[#Thread_Dump|Thread Dump]] and restart.

== Gerrit metrics ==

On top of the JavaMelody data, [[gerrit:Documentation/metrics.html|Gerrit has internal metrics]].

For users having the [[gerrit:Documentation/access-control.html#capability_viewCaches|| viewCaches]] or [[gerrit:plugins/metrics-reporter-prometheus/Documentation/config.md|View Metrics]] capabilities, various internal Gerrit metrics can be retrieved via:

* https://gerrit.wikimedia.org/r/config/server/metrics

Which obviously requires authentication. That complements &lt;code&gt;gerrit show-caches&lt;/code&gt;.

We use the metrics-reporter-prometheus plugin which exposes  collected by Prometheus from https://gerrit.wikimedia.org/r/plugins/metrics-reporter-prometheus/metrics . Those Gerrit metrics can also be seen on [https://gerrit.wikimedia.org/r/monitoring?part=mbeans the JavaMelody MBeans page] under the &lt;code&gt;metrics&lt;/code&gt; branch.

See [https://grafana.wikimedia.org/dashboards/f/5AnaHr2Mk Gerrit Grafana dashboards folder].

== Use &lt;code&gt;gerrit ssh&lt;/code&gt; commands ==
To use Gerrit ssh commands (see [https://gerrit-review.googlesource.com/Documentation/cmd-index.html#user_commands command list documentation]), you will need to be a member of the ldap group gerritadmin. To do so, (more info on the [[SRE/LDAP]] documentation), you will need to run &lt;code&gt;ldapvi -b ou=groups cn=gerritadmin&lt;/code&gt; and then wait for the synchronization to happen. Then, you'll be able to run commands such as [[Gerrit/Operations#Killing ssh connections]] or &lt;code&gt;ssh user@gerrit.wikimedia.org -p 29418 gerrit show-caches&lt;/code&gt; etc.
{{Warn
| content = Please avoid running replication command without asking RelEng or Collab first (see https://wikitech.wikimedia.org/wiki/Gerrit/Administration#Forcing_Replication_re-runs)
}}

== Logs ==

They are consumed by our logging infrastructure and available in the [https://logstash.wikimedia.org/app/dashboards#/view/AW1f-0k0ZKA7RpirlnKV Kibana dashboard for Gerrit (application logs)] and [https://logstash.wikimedia.org/goto/e79e3841a2cd0cc17ed16481bd5fb1da Apache access logs].

=== Main logs ===
Logs are available on the gerrit servers at: &lt;code&gt;/var/log/gerrit/&lt;/code&gt;. There are a number of logfiles:

* &lt;code&gt;gerrit.log&lt;/code&gt;: This is the main log file and will show stacktraces and errors
* &lt;code&gt;gerrit.json&lt;/code&gt;: Like &lt;code&gt;gerrit.log&lt;/code&gt; bug not really human readable. For sending structured logs to logstash.
* &lt;code&gt;sshd_log&lt;/code&gt;: Log of sshd events
* &lt;code&gt;gc_log&lt;/code&gt;: Logs for &lt;code&gt;git gc&lt;/code&gt; '''not''' the JVM garbage collection (those logs are available in &lt;code&gt;/srv/gerrit/jvmlogs&lt;/code&gt;)
* &lt;code&gt;plugin_log&lt;/code&gt;: Info about plugins being loaded and reloaded, this information is also in &lt;code&gt;gerrit.log&lt;/code&gt;

=== HTTP Logs ===

Gerrit sits behind Apache, access and error logs are both in &lt;code&gt;/var/log/apache2&lt;/code&gt;:

* &lt;code&gt;gerrit.wikimedia.org.https.access.log&lt;/code&gt;
* &lt;code&gt;gerrit.wikimedia.org.https.error.log&lt;/code&gt;
find its logs by searching with &lt;code&gt;type:log4j&lt;/code&gt;.

== JVM ==

=== Thread Dump ===

A thread dump is often useful in troubleshooting. To capture a thread dump use &lt;code&gt;jstack&lt;/code&gt;. This code should be safe to run at any time, and is run frequently while Gerrit is running:

 sudo -u gerrit2 jstack -l $(pgrep java) &gt; &quot;/srv/gerrit/jstack-$(date +%Y-%m-%d-%H%M%S).dump&quot;

It's often useful to upload the resulting file to https://fastthread.io/ to detect problems.

=== Java trace ===
{{warn|This command isn't run very often, unsure how safe it is to run; kept here for folks who are familiar with jstat}}

Display a summary of garbage collection statistics every 1000 ms:

 sudo -u gerrit2 /usr/lib/jvm/java-8-openjdk-amd64/bin/jstat -gcutil &quot;$(pgrep -u gerrit2 java)&quot; 1000

=== Java heap usage ===

''Requires openjdk-X-dbg for the debugging symbols''

   sudo /usr/lib/jvm/java-8-openjdk-amd64/bin/jmap -heap &quot;$( pgrep -u gerrit2 java)&quot;

== Access h2 account_patch_reviews ==

On copies of account_patch_reviews* files:
&lt;pre&gt;
java -cp h2-1.3.176.jar org.h2.tools.Shell -url jdbc:h2:/home/hashar/account_patch_reviews
&lt;/pre&gt;
Which gives you a sql prompt:
&lt;pre&gt;
sql&gt; show columns from ACCOUNT_PATCH_REVIEWS
...&gt; ;
FIELD        | TYPE         | NULL | KEY | DEFAULT
ACCOUNT_ID   | INTEGER(10)  | NO   | PRI | 0
CHANGE_ID    | INTEGER(10)  | NO   | PRI | 0
PATCH_SET_ID | INTEGER(10)  | NO   | PRI | 0
FILE_NAME    | VARCHAR(255) | NO   | PRI | ''
(4 rows, 16 ms)
&lt;/pre&gt;

== Blocking misbehaving bots / IPs ==

If necessary either IP addresses or user agents that are misbehaving can be blocked by making edits to '''modules/profile/templates/gerrit/apache.erb''' in the '''operations/puppet''' public git repository
and merging them.

[https://gerrit.wikimedia.org/r/c/operations/puppet/+/1037856/5/modules/profile/templates/gerrit/apache.erb example change]

=== Throttling IPs ===

Since September 2024, implemented in [[phab:T365259]] there is another method of throttling abusive traffice using [[nftables]]. See [[Firewall#Throttling_with_nftables]] and the [https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/gerrit.yaml profile::firewall::nftables_throttling keys in Hiera].

You can also observe data related to this on the [https://grafana.wikimedia.org/goto/iU0RPFiHR?orgId=1 grafana dashboard for gerrit].

== Killing ssh connections ==

It can happen that a user reaches the limit of 8 concurrent ssh connections and then says they can't push to Gerrit anymore over ssh.

A member of Gerrit admins can run commands like these to kill connections for them:


 ssh user@gerrit.wikimedia.org -p 29418 gerrit show-connections
 ssh user@gerrit.wikimedia.org -p 29418 gerrit close-connection &lt;connection ID&gt;

== Failover (in case primary host is unavailable) ==

# Disable puppet on the elected replacement
# Enable read only with the cookbook&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
sudo cookbook sre.gerrit.read-only-toggle --host gerrit1003 --toggle on
&lt;/syntaxhighlight&gt;
# Run the local_backup cookbook ([[Gerrit/Operations#Warming up local backup|Gerrit/Operations#Warming_up_local_backup)]]&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
sudo cookbook sre.gerrit.localbackup --source gerrit1003
&lt;/syntaxhighlight&gt;
# Switch role locally on the gerrit systemd unit with &lt;syntaxhighlight lang=&quot;shell&quot;&gt;
systemctl edit gerrit.service --full
&lt;/syntaxhighlight&gt;
## Remove &lt;code&gt;--replica&lt;/code&gt; from &lt;code&gt;ExecStart=&lt;/code&gt;
## Remove &lt;code&gt;--enable-httpd&lt;/code&gt; from &lt;code&gt;ExecStart=&lt;/code&gt;
## Save and quit
## Run &lt;code&gt;systemctl daemon-reload&lt;/code&gt;
# Switch the &lt;code&gt;CNAME&lt;/code&gt; discovery record in our DNS repo in [[gerrit:plugins/gitiles/operations/dns/+/refs/heads/master/templates/wmnet#1044|templates/wmnet#1044]] as described in [[phab:T387833#11601246|this task]] to designate the newly promoted replacement. For this you'll need to follow [[DNS#Emergency Measures]] 
# Restart gerrit with &lt;code&gt;systemctl restart gerrit &amp;&amp; journalctl -fln50 -u gerrit&lt;/code&gt;
# Confirm that you can now access the [https://gerrit.wikimedia.org new primary] and clone projects both via SSH and HTTPS&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
git clone --depth=1 https://gerrit.wikimedia.org/r/mediawiki/extensions.git
git clone --depth=1 &quot;ssh://gerrit.wikimedia.org:29418/mediawiki/extensions&quot;

&lt;/syntaxhighlight&gt;
# Disable read only mode:&lt;syntaxhighlight lang=&quot;shell&quot;&gt;
sudo cookbook sre.gerrit.read-only-toggle --host gerrit1003 --toggle off
&lt;/syntaxhighlight&gt;
# Backport the changes to puppet
# Resume replication

== Switch replicas over ==


In [[phab:T406334|T406334]] we switched replicas, editing DNS and Puppet as follows:

* [[gerrit:c/operations/dns/+/1242268|operations/dns/+/1242268]] (and its hotfix [[gerrit:c/1243119|1243119]])
* [[gerrit:c/operations/puppet/+/1242269|operations/puppet/+/1242269]] 


== Switchover (planned maintenance) ==
{{Infobox2|data1=If you're migrating on gerrit2003, please see https://phabricator.wikimedia.org/T338470#10506291 
Try to ensure the user running Gerrit will be also owning the synced data.}}
=== Schedule and Announce Downtime ===
Announce the scheduled downtime for Gerrit services. 

{{Warning
| content = Please also take note of that specific section of Gerrit's administration page: https://wikitech.wikimedia.org/wiki/Gerrit/Administration#Forcing_Replication_re-runs
}}

=== Prepare patches ===
Those hiera keys have to be updated:
 profile::gerrit::active_host
 profile::gerrit::replica_hosts
 profile::gerrit::lfs_sync_dest
(see [[gerrit:c/operations/puppet/+/1217133|example patchset]]){{Warn
| content = You can submit those patches but do not merge them.
}}
You also have to update the [[DNS|DNS configuration]] (see [[gerrit:c/operations/dns/+/1238708|example patchset]]).

=== Automated switchover ===
{{Caution|Be aware that warming up the transfers will erase all data on the target instance.}}

==== Warming up transfer ====
 sudo cookbook sre.gerrit.sync-instances --source gerrit1003 --replica gerrit2003 --chown --distrust
The args &lt;code&gt;--distrust&lt;/code&gt; and &lt;code&gt;--chown&lt;/code&gt; are given to bypass Gerrit internal replication, this will change in the future. Those arguments imply we run the &lt;code&gt;rsync&lt;/code&gt; commands listed below in the manual switchover procedure via the cookbook and then ensure ownership of the transferred files. 

==== Warming up local backup ====
 sudo cookbook sre.gerrit.localbackup --source gerrit1003
Here, we run a local backup to ensure having a local (on the source instance) snapshot of our data before doing anything.

==== Switching Over ====
 sudo cookbook sre.gerrit.switchover --switch-from-host gerrit1003 --switch-to-host gerrit2003 --distrust --chown
The &lt;code&gt;--distrust&lt;/code&gt; and &lt;code&gt;--chown&lt;/code&gt; args are also required here to be passed on to the other cookbooks that will be used to perform the switchover. The cookbook will guide you through the next steps, asking you to merge the Puppet patch and then the DNS patch.

=== Manual switchover ===

==== Prepare emergency rollback commands for DNS ====
Preparing the commands listed in [[DNS#Update DNS if Gerrit is down]] could help revert if needed.

==== Stop Puppet across all gerrit instances ====
 sudo disable-puppet 'gerrit maintenance'

==== Merge the puppet changes to prepare the switch over to &lt;code&gt;dst-gerrit&lt;/code&gt; ====
Merge the Puppet changes to prepare for the desired state.

==== Begin Scheduled Downtime ====
Announce the start of the scheduled downtime on IRC &lt;code&gt;#wikimedia-operations&lt;/code&gt; and on Slack &lt;code&gt;#engineering-all&lt;/code&gt;.

==== Downtime Management ====
 sudo cookbook sre.hosts.downtime -r 'maintenance' -D 30 src-gerrit.wikimedia.org &amp;&amp; sudo cookbook sre.hosts.downtime -r 'maintenance' -H 1 dst-gerrit.wikimedia.org
Manually schedule downtime for checks connected to the virtual server &quot;&lt;code&gt;gerrit.wikimedia.org&lt;/code&gt;&quot; on &lt;code&gt;icinga.wikimedia.org&lt;/code&gt;.

==== Update DNS ====
[[DNS#Deploying DNS changes|Run]] &lt;code&gt;sudo -i authdns-update&lt;/code&gt; on &lt;code&gt;ns0.wikimedia.org&lt;/code&gt;, review the diff '''but do not commit yet'''.

==== Stop Gerrit on &lt;code&gt;dst-gerrit&lt;/code&gt; ====
Execute the following commands on &lt;code&gt;dst-gerrit&lt;/code&gt;:
 sudo systemctl stop gerrit

==== Make source and destination read-only ====
Execute the following commands on '''both instances''':
 touch /etc/gerrit/gerrit.readonly
That will disable write operation and offer some safety from data corruption

==== Data Synchronization from &lt;code&gt;src-gerrit&lt;/code&gt; ====
 rsync -avpPz --delete /var/lib/gerrit2/review_site/ rsync://dst-gerrit.wikimedia.org/gerrit-var-lib/
 rsync -avpPz --delete /srv/gerrit/ rsync://dst-gerrit.wikimedia.org/gerrit-data/ --exclude=*.hprof

==== Stop Gerrit on &lt;code&gt;src-gerrit&lt;/code&gt; ====
Execute the following commands on &lt;code&gt;src-gerrit&lt;/code&gt;:
 sudo systemctl stop gerrit

==== Repeat Data Synchronization on &lt;code&gt;src-gerrit&lt;/code&gt; ====
Repeat the rsync commands as in this [[Gerrit/Operations#Data Synchronization on src-gerrit|step]].

This is to ensure consistency across nodes.

==== Start Gerrit on &lt;code&gt;dst-gerrit&lt;/code&gt; ====
Execute the following command on &lt;code&gt;dst-gerrit&lt;/code&gt;:
 sudo systemctl start gerrit

==== Finalize DNS Update ====
Confirm the DNS change and merge it.

==== Finalize Puppet Update ====
You can also speed-up the subsequent deployment by running &lt;code&gt;sudo run-puppet-agent&lt;/code&gt; on &lt;code&gt;dst-gerrit&lt;/code&gt;. 

=== Testing ===
* HTTPS access via browser: https://gerrit.wikimedia.org
* HTTPS access via browser: https://gerrit-replica.wikimedia.org
* SSH access: &lt;code&gt;ssh '''''yourusernameongerrit'''''@gerrit.wikimedia.org -p 29418&lt;/code&gt;

=== Announce Downtime Conclusion ===
Announce that the downtime is over.

=== Post-Migration Tasks ===

# Check if replication is running: &lt;code&gt;ssh gerrit.wikimedia.org -p 29418 gerrit show-queue --by-queue --wide&lt;/code&gt;
# Validate that https://gerrit-replica.wikimedia.org/ returns a HTTP/404 error on &lt;code&gt;/&lt;/code&gt;
# Determine the grace period duration.
# Ensure &lt;code&gt;src-gerrit&lt;/code&gt; has Puppet disabled and/or services are masked, if it needs to be decommissioned.
# If needed, decommission the old host as per [[phab:T336427|T336427]].

== Runbooks ==

=== GerritHAProxyBackendUnavailable ===
Gerrit service is not available to the tcp-proxy hosts. Most likely the active Gerrit server is down and the tcp-proxy hosts can not reach the active Gerrit server anymore. Maintenance on the active Gerrit server could be the issue and a downtime is missing for the &lt;code&gt;tcp-proxy*&lt;/code&gt; hosts.

If there is no planned maintenance the Gerrit active server should be troubleshooted (see steps above), make sure the &lt;code&gt;gerrit&lt;/code&gt; and &lt;code&gt;apache2&lt;/code&gt; and &lt;code&gt;envoy&lt;/code&gt; services are running. Also network connectivity from the tcp-proxy hosts should be possible.

=== GerritHAProxyServiceUnavailable ===
Gerrit tcp-proxy service is not available in at least one PoP and there is less than one available tcp-proxy per DC. Most likely the active Gerrit server is down and the tcp-proxy hosts can not reach the active Gerrit server anymore so they are marked as &lt;code&gt;DOWN&lt;/code&gt;. Also a PoP might be depooled but not properly downtimed.

If there is no planned maintenance the network connectivity from the tcp-proxy hosts to the Gerrit active server should be troubleshooted and make sure the &lt;code&gt;gerrit&lt;/code&gt; and &lt;code&gt;apache2&lt;/code&gt; and &lt;code&gt;envoy&lt;/code&gt; services are running.</text>
      <sha1>cy9neoje2rde8u7f8gckwnzzz2dy2n9</sha1>
    </revision>
  </page>
  <page>
    <title>Annual Toolforge Survey/How to</title>
    <ns>0</ns>
    <id>448709</id>
    <revision>
      <id>2414321</id>
      <parentid>2412228</parentid>
      <timestamp>2026-05-16T22:02:51Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414321</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="3720" sha1="7q9wnktgz0d4981jjvd0oy7lwdkwyoq" xml:space="preserve">{{Historical}}
The current process can be found at [[Annual_Toolforge_Survey/Ops]] 
= Steps =

# Create a phabricator task for the survey (such that there's one task per year).
# Draft and review the message body for the survey announcement.
# Copy, update, review the survey form. The survey lives on LimeSurvey.
# Publicly announce survey (typically on a Monday in November or early December).
# One week later remind with &quot;[Reminder]&quot; prepended to subject.
# About a day before the survey ends remind with &quot;[Final reminder]&quot; prepended to the subject.

= Announcement template =
&lt;pre&gt;
Subject: Participate in the Wikimedia Cloud Services Annual Survey (2021)

Hello!

You are receiving this email because you are a member of the Toolforge project or you are a project admin for a Wikimedia Cloud VPS project.

We, the Wikimedia Cloud Services team, are conducting this survey to learn more about how developers use our services and how they can be improved to address their needs. This survey will take between 10 minutes and an hour of your time, depending on how much feedback you would like to share with us in the form of free form comments.

This survey will be conducted via a third-party service, which may subject it to additional terms. For more information on privacy and data-handling, see the survey privacy statement [0].

If you agree to these terms and conditions, please go to https://wikimediafoundation.limesurvey.net/565681 to participate in the survey. 
The survey will end on Tuesday, Dec 20th, 2022.

If you do not wish to receive future Cloud Services annual survey emails, add your Wikimedia developer account name (i.e. the username you use to login to wikitech.wikimedia.org) to the opt-out list [1].

If you want to learn more about the survey and its result, please visit: &lt;https://meta.wikimedia.org/wiki/Research:Cloud_Services_Annual_Survey/2021&gt;

[0]: https://foundation.wikimedia.org/wiki/Legal:2022_Wikimedia_Cloud_Services_Survey_Privacy_Statement
[1]: https://wikitech.wikimedia.org/wiki/Annual_Toolforge_Survey/Opt_out

&lt;/pre&gt;

= Reminder template =
&lt;pre&gt;
Subject: 1 week left - Participate in the Wikimedia Cloud Services Annual Survey (2021)

Reminder, there is only one week left to fill the survey. We would really appreciate your feedback.

Wikimedia Cloud Services team, is conducting this survey to learn more about
how developers use our services and how they can be improved to address their needs.
We would really appreciate your feedback.

If you have already filled out the survey, thank you!


* Privacy policy: https://foundation.wikimedia.org/wiki/Legal:2022_Wikimedia_Cloud_Services_Survey_Privacy_Statement
* Survey Form: https://wikimediafoundation.limesurvey.net/565681
* Opt out: https://wikitech.wikimedia.org/wiki/Annual_Toolforge_Survey/Opt_out

&lt;/pre&gt;


= Bulk emailing instructions =
Tools for generating the survey list can be found at https://gitlab.wikimedia.org/repos/cloud/wmcs/wmcs-survey-mail-list
&lt;p&gt;The scripts should be run on a `cloudweb` host.

&lt;pre&gt;
Make a list of usernames suitable for surveying all Toolforge
maintainers + Cloud VPS project admins.

$ python3 make-toolforge-user-list.py &gt; users.txt
$ python3 make-cloudvps-user-list.py &gt;&gt; users.txt
$ sort users.txt | uniq &gt; all-users-sorted.txt
&lt;/pre&gt;

Once the list of recipients is complete, send emails using https://www.mediawiki.org/wiki/Manual:SendBulkEmails.php.  Be sure to specify --optout https://wikitech.wikimedia.org/wiki/Annual_Toolforge_Survey/Opt_out (eg. --optout &quot;Annual_Toolforge_Survey/Opt_out&quot;)

Note that some users might reach out via email asking to opt out of receiving the survey. You will have to manually add them to the opt out list on the wiki page yourself.</text>
      <sha1>7q9wnktgz0d4981jjvd0oy7lwdkwyoq</sha1>
    </revision>
  </page>
  <page>
    <title>Map of database maintenance</title>
    <ns>0</ns>
    <id>449160</id>
    <revision>
      <id>2414332</id>
      <parentid>2414307</parentid>
      <timestamp>2026-05-17T00:02:37Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2414332</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="841" sha1="fqk67wspqz3m50u404hu3153qb7pxmo" xml:space="preserve">{{/Header}}
== Today (2026-05-17) ==
== Yesterday (2026-05-16) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| es6 || [[phab:T419961|Login (T419961)]] (fceratto)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| pc3 || [[phab:T418973|Productionize pc20[21-24] and pc10[21-24] (T418973)]] (marostegui)
|-
| s7 || 
* [[phab:T419961|Login (T419961)]] (fceratto)
* [[phab:T426142|Switchover s7 master (db2220 -&amp;gt; db2218) (T426142)]] (marostegui)
* [[phab:T426380|Switchover s7 master (db2218 -&amp;gt; db2220) (T426380)]] (marostegui)
|-
| s8 || 
* [[phab:T419635|Drop il_to column from imagelinks table in wmf production (T419635)]] (fceratto)
* [[phab:T419961|Login (T419961)]] (fceratto)
* [[phab:T426291|Switchover s8 master (db2165 -&amp;gt; db2161) (T426291)]] (fceratto)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>fqk67wspqz3m50u404hu3153qb7pxmo</sha1>
    </revision>
  </page>
  <page>
    <title>Test Kitchen/FAQ</title>
    <ns>0</ns>
    <id>451360</id>
    <revision>
      <id>2414323</id>
      <parentid>2403588</parentid>
      <timestamp>2026-05-16T22:08:27Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414323</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="21394" sha1="dtt08j6xd1yf850ptwhksawfsqt9zls" xml:space="preserve">This page includes information about the purpose, functionality, and status of Test Kitchen.

== General FAQs ==

=== What core capabilities does the Test Kitchen deliver? ===
# Simpler approach and tools for [[Test Kitchen/Create_an_instrument|on-wiki instrumentation]]
# [[Test Kitchen/Conduct an experiment|Coordination]] and [[Test Kitchen/Automated analysis of experiments|analysis]] of experiments (A/B tests)
# Off-wiki instrumentation support

=== What are the benefits of Test Kitchen? ===
Test Kitchen provides standardized tools and processes to:

* Reduce experiment setup time from 10 weeks to 1 week
* Enable testing across multiple wikis, languages, and platforms
* Support both registered, temporary and [[Edge uniques|anonymous user testing]]
* Automate data collection and analysis
* Ensure compliance with [[foundation:Policy:Privacy_policy|privacy policies]] and security requirements
* Make experiment results publicly accessible to movement stakeholders

=== What's the best route for an engineer on my team, who has not used Test Kitchen before, to get familiar and able to use it for an upcoming experiment? ===
Start by identifying which workflow fits your needs: [[Test Kitchen/Conduct an experiment|Conduct an experiment]] if you're running an A/B test, or [[Test Kitchen/Measure product health|Measure product health]] if you're setting up baseline instrumentation. These pages will walk you through the process end to end.

Follow the [[Test Kitchen/Local development setup|Local development setup]] guide to get your environment ready.

If you get stuck or want feedback on your approach, the #talk-to-experiment-platform Slack channel is the best place to ask questions. Sharing your measurement plan ([https://docs.google.com/document/d/1Kw2xTnX1lu-0dxVb667IjfKgI17fVXiVyTVVgzA8yBo/edit?tab=t.0 template]) or instrumentation spec ([https://docs.google.com/spreadsheets/d/1e4XiPmetbKqXEvQQIBICUiK2H-r0BlWQrb0YJRHVJDw/edit?gid=1635536298#gid=1635536298 template]) will help us give you useful guidance.

== Test Kitchen &amp; Events ==

=== How Does Test Kitchen Relate to the Event Platform? ===
Test Kitchen may be thought of as an &quot;opinionated&quot; [[Event Platform]] client – and because it's opinionated there's less for you to do. It does not replace the Event Platform; it works alongside it, making instrumentation accessible to engineers and community members who don't have Data Engineering support.

Firstly, Test Kitchen owns and maintains a set of [[Test Kitchen/Web schema|base schemas]] with which your events will be validated, so that you do not have to create a new schema for each new instrument. These schemas include properties for the most common instrument-agnostic data that teams might need to answer their questions, e.g. session ID, pageview ID, namespace and title of the current page. The base schemas also have properties that can hold instrument-specific [[Test Kitchen/Interaction data|interaction data]].

Secondly, your code passes event names and data to Test Kitchen, rather than streams and events. That is, rather than writing an instrument that ''submits'' events to a specific stream, you write an instrument that ''dispatches'' events, constructed by Test Kitchen from your event data, to zero or more ''interested'' streams.

If you need custom data not available in the base schemas, you can create a [[Test Kitchen/Custom schemas|custom schema]] that references them. Note that you are responsible for owning and maintaining any custom schemas you create.

=== Will it replace the current event platform? ===
'''No.''' Test Kitchen represents a new model of data collection to make the instrumentation process faster and easier to use. It is a client of the Event Platform itself. It's designed to make instrumentation accessible to engineers and community members who don't have Data Engineering support, by removing the need to create schemas and manage stream configuration manually.

== On-Wiki Instrumentation ==

=== What will be different if I use Test Kitchen for instrumentation? ===
'''Key Differences:'''

# There’s no need to deploy a new schema. Instrumentation happens in the codebase in accordance with the Test Kitchen SDK's rules.
# Enablement of data collection from an instrument is performed by the Feature Engineer in consultation with an Analyst.
# All configuration of event streams happens in the control plane.
# Data Models can be applied to event streams directly for faster and more flexible querying.
{| class=&quot;wikitable&quot;
|+Workflow Steps Using Test Kitchen &amp; Current Process
!'''Process Steps'''
!'''Test Kitchen'''
!'''Legacy Process'''
|-
|Teams Involved
|1-2
|3-4
|-
|Number of steps to start collecting data
|3
|10-12
|-
|Time needed to start collecting data
|1 day post code deploy
|6-10 weeks
|-
|Requires schema development
|No
|Yes
|-
|Can be included in volunteer projects
|Yes
|No
|-
|Automatic data collection termination 
|Yes
|No
|-
|Data Collection Guideline support
|Yes
|No
|}

=== Can I still use the existing process for instrumentation? ===
'''Yes'''. Backwards compatibility with the current Event Platform is and will be maintained if you and your team prefer to use this process.

=== Will I be forced to migrate all my existing schemas and instrumentation to Test Kitchen? ===
'''No'''. Backwards compatibility with the current Event Platform is and will be maintained and so your existing data will continue to be collected.

=== What is the process for creating new instrumentation? ===
Follow this guide for [[Test Kitchen/Measure product health|measuring product health]].  

=== If I migrate existing instruments, does that impact existing Analytics scripts, visualisations, etc.? ===
Only if you decommission your old instrumentation. You have two paths: 

* Run both in parallel: if you leave the existing instrumentation in place, running alongside Test Kitchen, nothing will break. 
* Migrate fully: if you chose to deprecate existing instrumentation and re-instrument using Test Kitchen, you will need to change your queries to match your new data model. Your data model can result in the exact same output if desired - in which case you would just need to change the table names where appropriate. 

=== How do we know that data quality is consistent with existing instrumentations? ===
Because Test Kitchen is itself an Event Platform (EP) client, you can expect the same baseline event rate and data quality regardless of whether you use Test Kitchen or not. We are always running experiments&lt;ref&gt;https://phabricator.wikimedia.org/T401453&lt;/ref&gt;&lt;ref&gt;https://phabricator.wikimedia.org/T420738&lt;/ref&gt; to test, measure and improve data quality. 

Test Kitchen provides the same common contextual attributes as existing instrumentation, and does not modify instrumentation-specific data that it is passed.

== Experimentation &amp; Feature Flags ==

=== What is experimentation and why does it matter? ===
Experimentation, or [[A/B testing]], is how we test the actual impact of product changes with our users. When we make product decisions, we’re acting on assumptions – that a change will make something easier, more useful, or more engaging. Experimentation lets us test ''if'' those assumptions are correct ''before'' committing to them.  

This matters because our assumptions are frequently wrong. Without experimentation, we risk making confident decisions based on intuition that the data would have contradicted.

Test Kitchen provides a unified platform for running experiments consistently across wikis, languages, and platforms. This consistency means results are comparable across teams, best practices are encoded by default, and teams can run experiments without needing specialist support for each one.

[[mw:Why_we_experiment|Learn more]] about why and how we experiment at the Wikimedia Foundation.

=== What is a feature flag? ===
Features Flags enable you to change your products behavior from a central location without requiring an entirely new deployment. For example, turn on/off a change to a toolbar or change the placement of buttons in a UI. Engineers and PMs can set a global value for everyone, use traffic rules to assign values to user demographics, and run experiments between different implementations of a feature.

== Experiment Design ==

=== What's the difference between using &lt;code&gt;mw-user&lt;/code&gt; and &lt;code&gt;edge-unique&lt;/code&gt; as my identifier type, and how do I choose? ===
These identifier types determine who is enrolled in your experiment and how enrollment is managed.

&lt;code&gt;mw-user&lt;/code&gt; enrolls users via their CentralAuth global ID. Enrollment is consistent across all wikis and sessions — a user who logs out and back in will remain in the same experiment group. This includes temporary accounts. Use &lt;code&gt;mw-user&lt;/code&gt; when you need reliable data about logged-in user behavior.

&lt;code&gt;edge-unique&lt;/code&gt; enrolls clients via an [[Edge uniques|anonymous cookie (wmf-uniq)]]. Enrollment covers all user traffic — logged-in, temporary and logged-out — but is based on the cookie rather than user identity. If the cookie is cleared, the client may be re-enrolled into a different group. Note that experiments using edge-unique can only use client-side instrumentation — MediaWiki does not have access to the wmf-uniq cookie. Use edge-unique when your experiment targets logged-out users.

Because &lt;code&gt;edge-unique&lt;/code&gt; experiments include both logged-in and logged-out users, results should be analyzed separately by authentication status: logged-out, permanent, and temporary accounts. If you need reliable insights specifically about logged-in user behavior, we recommend running a separate &lt;code&gt;mw-user&lt;/code&gt; experiment in parallel or as a follow-up.

For more details, see [[Test Kitchen/Conduct an experiment#Experiment design: identifier type|Experiment design: identifier type]].

Note: A logged-in user enrolled in both an &lt;code&gt;edge-unique&lt;/code&gt; and an &lt;code&gt;mw-user&lt;/code&gt; experiment may fall out of the &lt;code&gt;edge-unique&lt;/code&gt; experiment if their cookies are cleared. Their &lt;code&gt;mw-user&lt;/code&gt; enrollment will not be affected.

Note: A user enrolled in an &lt;code&gt;edge-unique&lt;/code&gt; experiment who then creates an account will still be enrolled in the experiment after account creation, because edge uniques work for logged out, permanent, and temporary account users. 

=== Is &lt;code&gt;experiment.subject_id&lt;/code&gt; consistent for a logged-in user across platforms and wikis? Is it consistent for both logged-in and logged-out users? ===
Yes and yes! When using &lt;code&gt;mw-user&lt;/code&gt;, the user will be consistently enrolled and assigned across all wikis and across all their sessions. The user may log out and log back in, and their enrolment and assignment will not change.&lt;ref&gt;https://wikitech.wikimedia.org/wiki/Test_Kitchen/Conduct_an_experiment#Experiment_design:_identifier_type&lt;/ref&gt; When using &lt;code&gt;edge-unique&lt;/code&gt;, the user will be consistently enrolled and assigned within top-level domain (e.g. wikipedia.org) for the lifetime of their wmf-uniq cookie.&lt;ref&gt;https://wikitech.wikimedia.org/wiki/Test Kitchen/Conduct_an_experiment#Experiment_design:_identifier_type&lt;/ref&gt;&lt;sup&gt;,&lt;/sup&gt;&lt;ref&gt;https://meta.wikimedia.org/wiki/Edge_Uniques/FAQ#More_detailed_technical_answer&lt;/ref&gt;
=== How many wikis and/or users would we need to achieve statistical significance? ===
'''The goal should never be to achieve statistical significance.''' Statistical significance (p-value) only tells you how likely you are to observe an impact as large or larger than what you observed if there were actually no real impact.

'''You're likely thinking of statistical power:''' the probability of correctly detecting an impact when one actually exists. While it's possible to run an underpowered experiment, if the true effect is small enough to be hard to detect, you won't be able to detect it reliably.

To conduct a proper power analysis, you need to know the baseline metric and the minimum effect size you want to detect. Since we lack historical data, we don't know realistic improvement targets (1%, 5%, 10%, or 20%). Running an experiment on enough wikis or for long enough could detect a tiny +0.01 percentage point change that's ''statistically'' significant at the 0.05 level, but this wouldn't represent a ''practically'' meaningful change.

When it comes to selecting wikis, ask: &quot;How generalizable are the results?&quot;

''The goal of experimentation is to produce results you can generalize from and use to predict the impact your intervention would have if rolled out to everyone.'' Therefore, '''determine a small, representative set of wikis (~20) that collectively represent your overall population''', since that's ultimately where you'll deploy the change. Focus on representativeness over raw numbers — a well-chosen sample of diverse wikis will give you more actionable insights than a large sample from similar wikis.

=== How does Test Kitchen handle running multiple experiments at the same time? ===
Test Kitchen is designed to support multiple concurrent experiments. Each experiment generates a unique subject ID by combining the unique identifier with the experiment name. As the number of experiments increases, the likelihood of being in more than one experiment increases.

To help you account for this, when analytics events are logged using &lt;code&gt;Experiment.send()&lt;/code&gt;, the system will soon ([[phab:T421152|T421152]]) automatically include an &lt;code&gt;experiment.other_assigned&lt;/code&gt; field. This records the assignments of any other experiments the user is currently enrolled in, so you can identify whether another experiment may have influenced your results.

=== What’s the best way to minimize collisions between experiments? ===
The best way to minimize collisions is to run experiments on different wikis. If two experiments are running on the same product surface for the same users, it becomes difficult to attribute changes in behavior to one experiment rather than the other — even if the features being tested are unrelated.

If you need to run experiments on the same wiki, check the &lt;code&gt;experiment.other_assigned&lt;/code&gt; field (after [[phab:T421152|T421152]]) in your analytics events to identify users who were enrolled in multiple experiments, and account for this in your analysis.

=== In an A/B/C test, when Test Kitchen divides 100% of a test group by 3 groups, where does the remainder go? ===
We use the [[gerrit:plugins/gitiles/mediawiki/extensions/TestKitchen/+/refs/heads/master/includes/Coordination/UserSplitterInstrumentation.php#39|hash of the user as a scaledHash (0 .. 1)]] and then we [[gerrit:plugins/gitiles/mediawiki/extensions/TestKitchen/+/refs/heads/master/includes/Coordination/UserSplitterInstrumentation.php#104|multiply this value by the number of buckets/groups]]. The result is the index of the bucket or group. That the remaining value will end up being part of the last treatment group.

If we have 2 buckets, the assignment, based on the calculated hash, would be something like the following:

* 0 - 0.5: group 1
* 0.5 - 1: group 2

And for three groups would be the following:

* 0 - 0.33: group 1
* 0.33 - 0.66: group 2
* 0.66 - 1: group 3

== Running an experiment ==

=== When we turn off an experiment, do events stop immediately? ===
Yes. The updated configuration propagates across all caching nodes in under 3 minutes. Once that's complete, the experiment is fully off: traffic is no longer split, no users are seeing treatments, and no data is being collected.

If you're considering ending an experiment before its planned date, make sure you have enough data to draw meaningful conclusions first. Stopping early, especially after seeing a promising early result, increases the risk of a false positive.&lt;ref&gt;https://thegood.com/insights/what-is-peeking/&lt;/ref&gt; If you're unsure whether you have sufficient data, ask in #talk-to-experiment-platform before turning it off.

=== How long should I run an experiment? ===
Run your experiment long enough to capture at least one full week of data, to account for day-of-week effects in user behavior. Beyond that, the right duration depends on your key metrics, as well as the minimum effect size you want to detect. An experiment with second-week retention as a key metric will take at least two weeks to collect the data needed, and smaller expected effects require longer run times to detect reliably.

As a general rule, '''decide on your intended run time before starting the experiment''' and stick to it. Stopping early because results look promising increases the risk of a false positive. If you're unsure how long to run your experiment, work with your Product Analyst or ask in #talk-to-experiment-platform. 

=== What kinds of changes are not suitable for experimentation? ===
'''Not every product decision needs an experiment.''' Experimentation is most valuable when you have a clear metric, a large enough user base to detect meaningful effects, and genuine uncertainty about whether the change will help.

* User research and usability testing are better suited to understanding why users behave the way they do, or to evaluating a design before it's built. If you're exploring a new concept or trying to identify pain points, talk to users before reaching for an experiment.
* Surveys are better suited to understanding user attitudes, preferences, and self-reported behavior — things that don't show up in behavioral data. If you want to know how users feel about a change, a survey will tell you more than an A/B test.
* Log analysis and observational data are better suited to understanding existing behavior patterns without introducing a change. If you're still forming your hypothesis, analyze what's already happening first.

A good experiment tests a specific, well-formed hypothesis. If you're not yet at that stage, another method will likely serve you better. Talk to your Product Analyst or ask in #talk-to-experiment-platform if you're unsure which approach fits your question.

=== Do metrics used in an experiment have to be in the Metrics Catalog? ===
No. But if your metric doesn't yet exist, you'll need to [[Test Kitchen/Automated analysis of experiments#Define metric|define it]], add it to the Metrics Catalog, and [[Test Kitchen/Automated analysis of experiments#Register experiment|register your experiment for automated analytics]]. Work closely with your Product Analyst to do so.

For example, a team wanting to measure &quot;the percent of users who choose three or more topics of interest&quot; could create a custom metric and add it to the catalog for reuse by other teams later. That said, we'd encourage teams to consider more continuous metrics where possible — such as average topics selected per user or topic selection completion rate — as these tend to produce more statistically robust results.

== Interpreting Results ==

=== What should I do if my experiment shows a negative result? ===
A negative or neutral result is a valid and valuable outcome: it tells you that your assumption was wrong, and saves you from investing further in a direction that doesn't help users. It is not a failure.

If your experiment shows a negative result, consider: Was the change implemented as intended? Was the experiment adequately powered to detect the effect you expected? Is there a subset of users for whom the change did have a positive effect that's worth exploring further?

Document your findings and share them internally and with our community. Negative results are just as important for the team's collective knowledge as positive ones. Ask in #talk-to-experiment-platform if you need help interpreting your results.

=== What do I do with my experiment results once I have them? ===
Once your experiment has run its course:

# Review results with your Product Analyst to ensure they're being interpreted correctly
# Make a ship/no-ship decision based on the results
# Document your decision and the reasoning behind it
# Share results with [[metawiki:List_of_experiments_in_Product_and_Technology|internal and movement stakeholders publicly]] on a project page
# Shift towards cleaning up the code

If you need a second opinion on your results or your interpretation, #talk-to-experiment-platform is the best place to ask.

== Aftermath ==

=== What do I do with the code when we've made our decision? ===
Once you've made your ship/no-ship decision, clean up promptly. Leaving experiment code in place longer than necessary adds technical debt and makes the codebase harder to maintain.

If you're shipping the change: remove the feature flag or toggle wrapping the treatment, make the treatment the default behavior

If you're not shipping: remove the treatment code entirely, Remove the instrument code that uses Test Kitchen SDK, remove the [[Test Kitchen/Stream configuration|stream configuration]], and remove the instrument from the [[Test Kitchen/Instrument list|instrument list]].

In both cases, follow the [[Test Kitchen/Decommission an instrument|Decommission an instrument]] guide to turn off data collection. If your experiment used a custom metric that won't be reused, consider whether it should be removed from the Metrics Catalog.

=== How do I decommission an experiment? ===
Follow the [[Test Kitchen/Decommission an instrument|Decommission an instrument]] guide

== References ==
&lt;references /&gt;</text>
      <sha1>dtt08j6xd1yf850ptwhksawfsqt9zls</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-review/SAL</title>
    <ns>498</ns>
    <id>452890</id>
    <revision>
      <id>2414342</id>
      <parentid>2414015</parentid>
      <timestamp>2026-05-17T08:58:11Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)</comment>
      <origin>2414342</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="28741" sha1="4cq3fk9qw4payg8ev8ln0l4agq2168k" xml:space="preserve">=== 2026-05-17 ===
* 08:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986406220 (https://github.com/cluebotng/component-configs/commits/be3bb145d2803394cd0b7dbd8ae1775ac9b7cd09)

=== 2026-05-14 ===
* 18:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970)

=== 2026-05-13 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b)

=== 2026-05-12 ===
* 10:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25727821525 (https://github.com/cluebotng/component-configs/commits/91aefb7d53013ad152bb721f71980dd26170f297)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842068 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 09:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724562210 (https://github.com/cluebotng/component-configs/commits/bd0e188642746ab949ec3762676ac730afff1c17)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/25723480598 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)
* 08:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723280529 (https://github.com/cluebotng/component-configs/commits/51d7c1919958a7672895885cbb3a1061934d2788)

=== 2026-05-06 ===
* 18:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25453754970 (https://github.com/cluebotng/component-configs/commits/92f164d1ab158aea1f76cd0a787f33ffe4017e85)

=== 2026-05-02 ===
* 12:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25252392660 (https://github.com/cluebotng/component-configs/commits/7352cd4f730ca9f5c276772f0b338230989feef4)

=== 2026-04-24 ===
* 22:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914278528 (https://github.com/cluebotng/component-configs/commits/23a4b53f3d291b0c750d44a2c0a661333307786d)

=== 2026-04-20 ===
* 23:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24695320410 (https://github.com/cluebotng/component-configs/commits/279edf060f43353ea66e6d057773bfdb883b16a1)

=== 2026-04-17 ===
* 00:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540678937 (https://github.com/cluebotng/component-configs/commits/26849735bbefbe218cbe0ce41db5a35941798c7b)

=== 2026-04-14 ===
* 21:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422836848 (https://github.com/cluebotng/component-configs/commits/10f4f0f81e169fac55d056176a273966c8160078)

=== 2026-04-11 ===
* 10:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280508671 (https://github.com/cluebotng/component-configs/commits/5953d3fb9c5e414df6995740382b7bd3be49ced2)
* 10:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24280372486 (https://github.com/cluebotng/component-configs/commits/5b34645dff3f37bc9f974635e03cd6b8436f37d1)

=== 2026-04-10 ===
* 23:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268430819 (https://github.com/cluebotng/component-configs/commits/3652893dce02243971055a6ab740363f103ce104)
* 23:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24268031378 (https://github.com/cluebotng/component-configs/commits/31367659ada078f50022f1df4b16b6139db27c09)
* 22:50 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24267603054 (https://github.com/cluebotng/component-configs/commits/ba252b54cec9387b47dd4ac4a347d4a9c5118c3e)
* 16:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:53 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24251572443 (https://github.com/cluebotng/component-configs/commits/2426c8db99c6d44c954ced07c9f41fcaa9e8e549)
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442439 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249898021 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:27 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247620609 (https://github.com/cluebotng/component-configs/commits/e63a941f5b83d97a9751af731c869062ceef4519)
* 14:26 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247581365 (https://github.com/cluebotng/component-configs/commits/49becfde53d5f960c8e4df0484cebb2bb4d4c5aa)
* 13:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24246395413 (https://github.com/cluebotng/component-configs/commits/945fa198e64a0e63b777bb570d57d68ef0ce3f69)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245816710 (https://github.com/cluebotng/component-configs/commits/6181fdda40150d3535541f3084ac7ff245f19536)
* 13:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245353001 (https://github.com/cluebotng/component-configs/commits/97eebf1bcdf5be901e0d3fd82c1b3ea6a8668163)
* 13:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24245099622 (https://github.com/cluebotng/component-configs/commits/251c10040c01caf2ba9b855050c318d5d2fd8e81)
* 13:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24244959273 (https://github.com/cluebotng/component-configs/commits/2a6605ee2d07c0ff0d690aaa8aabed0ca35bab72)
* 04:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24226442413 (https://github.com/cluebotng/component-configs/commits/4f895f83dae3f356cae2a1bbcfea51dd9d18bd15)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24221486227 (https://github.com/cluebotng/component-configs/commits/d96804861818d7786153d18d47be075a4dbbb6f2)

=== 2026-04-09 ===
* 20:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24212235180 (https://github.com/cluebotng/component-configs/commits/1cd21afab7312bd0122c0e735f8f4dca03019011)
* 19:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24208407251 (https://github.com/cluebotng/component-configs/commits/6cd680dd209bf7fbb01cf24cb6cca82f0fab716d)
* 18:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712296 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093220 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)
* 14:20 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24195131449 (https://github.com/cluebotng/component-configs/commits/6b512f6db7cc4e49078b135e437185906821ae81)

=== 2026-04-08 ===
* 05:03 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24118645397 (https://github.com/cluebotng/component-configs/commits/908dd70b5972cca0c0dafbe50a0020547b833a4e)

=== 2026-04-07 ===
* 22:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24106708812 (https://github.com/cluebotng/component-configs/commits/b85f56b6997ccf41cc8ea32f33a61809b68b9bc5)

=== 2026-04-02 ===
* 22:11 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9)

=== 2026-03-31 ===
* 06:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23782957628 (https://github.com/cluebotng/component-configs/commits/7888bbd75773dc064d78ad2ee8949f1540eab0fd)
* 01:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23775778726 (https://github.com/cluebotng/component-configs/commits/47f8e20c39e29b952f6dbbd04917970802ce1a0b)

=== 2026-03-27 ===
* 17:45 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659656811 (https://github.com/cluebotng/component-configs/commits/f4a494492433360a06326a918985c51c6d0828d4)
* 17:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537327 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-23 ===
* 10:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433070550 (https://github.com/cluebotng/component-configs/commits/dd92649311ee430b4225d5c6db5d6e6b16d10a86)
* 10:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23433054689 (https://github.com/cluebotng/component-configs/commits/c895b3d11b8546e54e4cca5ba350c0a5ca9c5917)

=== 2026-03-21 ===
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:23 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640335 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:19 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383565693 (https://github.com/cluebotng/component-configs/commits/48390b500ab2b65905e09987c12a3e42c3f69778)
* 16:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383551619 (https://github.com/cluebotng/component-configs/commits/ffff74b90a37a0c6bdd565128d3c11ae195e0763)

=== 2026-03-20 ===
* 04:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23329334324 (https://github.com/cluebotng/component-configs/commits/bd7700c30291bfab3a656aa8f257292e287a71ca)

=== 2026-03-19 ===
* 09:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23287809902 (https://github.com/cluebotng/component-configs/commits/0976850451c9fbb8c4afb773cc70b91cd7c6fdeb)

=== 2026-03-17 ===
* 21:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23217273642 (https://github.com/cluebotng/component-configs/commits/4fe6cb3d8ad39b60746b3b6bd2f83c4d05a82d6b)

=== 2026-03-13 ===
* 01:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23031229526 (https://github.com/cluebotng/component-configs/commits/ebd67e60183f161276bf0e13daab55ceb2463eb2)

=== 2026-03-10 ===
* 00:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22881598726 (https://github.com/cluebotng/component-configs/commits/bc32d8044077ff83db8b985b87df029ff564ad29)

=== 2026-03-07 ===
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22788149115 (https://github.com/cluebotng/component-configs/commits/b3731fab9a7f4f225ecbe318fa80808de6c904b0)

=== 2026-03-06 ===
* 09:08 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22756671151 (https://github.com/cluebotng/component-configs/commits/397fc33968a3c4795b97b1791a0b991ebeb81430)

=== 2026-03-04 ===
* 09:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22662750872 (https://github.com/cluebotng/component-configs/commits/01746ef8804c30c85963ea888a75887ebe879e3b)
* 01:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22650516003 (https://github.com/cluebotng/component-configs/commits/e7a1e2e06f2ccf038c06cb203369f336c298cf6c)

=== 2026-03-03 ===
* 21:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22642685900 (https://github.com/cluebotng/component-configs/commits/3cbfb68b3c0e7d97130ede1be762389f300234d2)

=== 2026-03-02 ===
* 01:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22557173037 (https://github.com/cluebotng/component-configs/commits/a414cf552e0a0c0d2c9e9817f922d56a4c899bf6)

=== 2026-02-27 ===
* 01:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22468544596 (https://github.com/cluebotng/component-configs/commits/b961f37db0544196a7206882b8e3f2292b7e0894)

=== 2026-02-25 ===
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22415923749 (https://github.com/cluebotng/component-configs/commits/a2a4f5ecffad1b49c96c33b5045430a5b75f71bc)
* 11:57 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22395760068 (https://github.com/cluebotng/component-configs/commits/c6093c4ed72aba8fa453b2f67e48d1effeaabb4b)

=== 2026-02-24 ===
* 21:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22370719324 (https://github.com/cluebotng/component-configs/commits/7170b95a5f9b6be3c928684f1e9c436deb3ddd1f)
* 00:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22331436693 (https://github.com/cluebotng/component-configs/commits/372e84511fdcb0893755ac22f399d3f24f438f7b)

=== 2026-02-21 ===
* 05:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22251332525 (https://github.com/cluebotng/component-configs/commits/29401fbb166f71e375eca7254fe841cb01836d2f)

=== 2026-02-20 ===
* 13:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22226385604 (https://github.com/cluebotng/component-configs/commits/fbd7d861a7062a2c09fd2117cbf569beb53916f4)
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22216991615 (https://github.com/cluebotng/component-configs/commits/64d521535aa35454c28900f70009efc0e9ff4a10)
* 05:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22212024170 (https://github.com/cluebotng/component-configs/commits/9b0508c1c5a875dd795c865e67f2a93d4f247597)

=== 2026-02-19 ===
* 13:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22183076688 (https://github.com/cluebotng/component-configs/commits/919ebb8860a93b9d071da361cad56448a3b1f2b4)
* 00:53 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22163899784 (https://github.com/cluebotng/component-configs/commits/0ce51e9bda73cc3ee0df647f7ba8dcfd02eb97e6)

=== 2026-02-16 ===
* 02:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22047780823 (https://github.com/cluebotng/component-configs/commits/3a1f6b151d38aab4ce1a62509b108ac9afc5230b)

=== 2026-02-15 ===
* 08:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22032600290 (https://github.com/cluebotng/component-configs/commits/80b9cda20d3f21e2f901db6ccbd168bfffb6b063)

=== 2026-02-14 ===
* 20:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023900270 (https://github.com/cluebotng/component-configs/commits/e8878c3f7a08aa1712126c1b6490f6db41621f44)
* 20:28 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22023724371 (https://github.com/cluebotng/component-configs/commits/de3779f7adea66769077e2380d7b0ce25f3d9e82)
* 20:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22023714306 (https://github.com/cluebotng/component-configs/commits/7cd082664340738b5c6cc46d0a195f3814672a3a)
* 19:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022830785 (https://github.com/cluebotng/component-configs/commits/9b3a727405218dd32b8f5b5d34d8906fe1ba840c)
* 19:16 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/22022827162 (https://github.com/cluebotng/component-configs/commits/0e1693c2b662aaa0c9264ceef355bcbfbc162ea7)
* 19:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022587196 (https://github.com/cluebotng/component-configs/commits/57dcc675b3ed54fc17f697d6c7b9554b5d06aab0)
* 18:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/22022444685 (https://github.com/cluebotng/component-configs/commits/1e0b17c59284d25ea8ac39a455abb9921ee6608a)

=== 2025-11-26 ===
* 19:58 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19715882591 (https://github.com/cluebotng/component-configs/commits/18c2bc79b5f0023e682a9245197cf87c5cc76943)

=== 2025-11-11 ===
* 15:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642915 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270263370 (https://github.com/cluebotng/component-configs/commits/d1674e8f4f6cec3b48e848137ce42585278d4a67)

=== 2025-11-09 ===
* 22:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215285730 (https://github.com/cluebotng/component-configs/commits/bf77359fc102b05a026ea8b66dc01ff16a936804)
* 22:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19215085734 (https://github.com/cluebotng/component-configs/commits/6d8f2491239fbe29d19544922253d9930a88e7a0)
* 20:30 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213996489 (https://github.com/cluebotng/component-configs/commits/38bc77281c9dbd1100915d95ba68705d8a7392a7)
* 20:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213940731 (https://github.com/cluebotng/component-configs/commits/c01b89b7b0455d4f1cc63a2eb002f9c55c0a663f)

=== 2025-11-05 ===
* 19:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19114484110 (https://github.com/cluebotng/component-configs/commits/fae01bfaeaeca0cf7676ece10cbd39948560086f)
* 16:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19108874377 (https://github.com/cluebotng/component-configs/commits/3f51ec3aa53d1378883a9dc973716e57c283d26c)

=== 2025-10-29 ===
* 15:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda)

=== 2025-10-23 ===
* 12:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18748267282 (https://github.com/cluebotng/component-configs/commits/bc8f1b883d0d53edf08bea5e5319ee7ee0b4fb82)

=== 2025-10-07 ===
* 06:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304366702 (https://github.com/cluebotng/component-configs/commits/5b83bca0e9293029698d7f3a1b2764727ae7f971)

=== 2025-10-06 ===
* 06:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272390911 (https://github.com/cluebotng/component-configs/commits/49abbdd5dd7066314199c213043305ceed2b54f7)

=== 2025-10-05 ===
* 06:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255184611 (https://github.com/cluebotng/component-configs/commits/7fe1a04069d9d0b4b11019443c85885c202852d4)

=== 2025-10-03 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18215013747 (https://github.com/cluebotng/component-configs/commits/7ab2bbe022e2513dc81a13a7055c4c7736e5f876)

=== 2025-09-29 ===
* 16:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2)
* 14:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc)
* 09:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)
* 09:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8)

=== 2025-09-27 ===
* 13:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060157048 (https://github.com/cluebotng/component-configs/commits/3aa079ed0cb7aa29f9ece46a47ad96203e53f242)

=== 2025-09-26 ===
* 22:18 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14)
* 11:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf)
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce)

=== 2025-09-25 ===
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c)
* 16:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc)
* 16:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 15:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c)
* 14:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0)

=== 2025-09-24 ===
* 17:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198575 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820837 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009157 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983740752 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17977211158 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 12:11 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976182663
* 12:06 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17976062312
* 06:46 wmbot~component-configs@tools-bastion: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17968643293

=== 2025-09-22 ===
* 20:43 wmbot~component-configs@tools-bastion: Test migrating log to feed channel
* 20:43 wmbot~damian-scripts@tools-bastion-15: Test migrating log to feed channel
* 19:12 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925793114

=== 2025-08-24 ===
* 18:02 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.3.0

=== 2025-08-23 ===
* 19:44 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.5
* 18:37 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.4
* 18:10 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.3

=== 2025-08-14 ===
* 13:39 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.2
* 13:27 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.1
* 13:20 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.2.0

=== 2025-08-13 ===
* 20:07 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.10
* 18:53 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.9

=== 2025-08-11 ===
* 19:49 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.7

=== 2025-08-10 ===
* 18:19 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.6
* 15:06 wmbot~damian-scripts@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.5

=== 2025-08-07 ===
* 15:38 wmbot~damian@tools-bastion-13: reviewer deployed @ refs/tags/v0.1.2

=== 2023-06-15 ===
* 18:43 wm-bot: &lt;root&gt; webservice restart, checked pods


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>4cq3fk9qw4payg8ev8ln0l4agq2168k</sha1>
    </revision>
  </page>
  <page>
    <title>Tool:Phab-ban/Log</title>
    <ns>116</ns>
    <id>453426</id>
    <revision>
      <id>2414337</id>
      <parentid>2411559</parentid>
      <timestamp>2026-05-17T01:27:41Z</timestamp>
      <contributor>
        <username>Phabbanbot</username>
        <id>37210</id>
      </contributor>
      <comment>Chicken.Tender.331 was disabled by Peachey88</comment>
      <origin>2414337</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="18702" sha1="rtzxgz1lseolqnuka47uknfnu0xuapj" xml:space="preserve">&lt;noinclude&gt;'''Audit log of bans''' made via https://phab-ban.toolforge.org. Some bans made prior to 2023-09-01 were manually logged at [[phab:T200856]]. __NOTOC____NOINDEX__&lt;/noinclude&gt;

=== 2026-05-17 ===
* 01:27 [[phab:p/Chicken.Tender.331|Chicken.Tender.331]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2026-05-07 ===
* 05:44 [[phab:p/Gabor_Kiss_WMSE|Gabor_Kiss_WMSE]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]

=== 2026-04-29 ===
* 22:31 [[phab:p/jhsoby-WMNO|jhsoby-WMNO]] was disabled by [[phab:p/Zabe/|Zabe]]
* 20:37 [[phab:p/Hr574380|Hr574380]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2026-04-22 ===
* 18:16 [[phab:p/Datronmcka|Datronmcka]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2026-04-14 ===
* 18:53 [[phab:p/Apokrif|Apokrif]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2026-04-07 ===
* 11:18 [[phab:p/Biof|Biof]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2026-03-15 ===
* 11:47 [[phab:p/Bucheon606|Bucheon606]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]
* 11:47 [[phab:p/Bucheon606|Bucheon606]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]
* 11:29 [[phab:p/Seoulbucheonincheon|Seoulbucheonincheon]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]
* 11:10 [[phab:p/Bucheonstation|Bucheonstation]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]
* 11:04 [[phab:p/WMFOfffce|WMFOfffce]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]
* 11:03 [[phab:p/WMFOfffce|WMFOfffce]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 11:03 [[phab:p/WMFOfffce|WMFOfffce]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]
* 10:57 [[phab:p/WMFOfflce|WMFOfflce]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]
* 10:29 [[phab:p/BucheonFac|BucheonFac]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]
* 09:48 [[phab:p/BucheonWest|BucheonWest]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 09:43 [[phab:p/TheBucheon|TheBucheon]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 09:40 [[phab:p/BucheonIncheon|BucheonIncheon]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]
* 09:33 [[phab:p/SkottishFinnishRadist|SkottishFinnishRadist]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 09:33 [[phab:p/SkottishFinnishRadist|SkottishFinnishRadist]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]
* 09:25 [[phab:p/BucheonCityHall6|BucheonCityHall6]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 09:16 [[phab:p/Primefac1|Primefac1]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 09:07 [[phab:p/ScottishFimishRadish|ScottishFimishRadish]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]
* 06:35 [[phab:p/Kgarcia181|Kgarcia181]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 04:27 [[phab:p/SldrF|SldrF]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 04:17 [[phab:p/PrimePac|PrimePac]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]
* 04:01 [[phab:p/260315t1244|260315t1244]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 03:34 [[phab:p/Bucheon543|Bucheon543]] was disabled by [[phab:p/DLynch/|DLynch]]
* 03:25 [[phab:p/LAG|LAG]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]
* 03:13 [[phab:p/Wonmidong|Wonmidong]] was disabled by [[phab:p/Novem_Linguae/|Novem_Linguae]]

=== 2026-03-14 ===
* 08:10 [[phab:p/Bucheon2026|Bucheon2026]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]
* 07:52 [[phab:p/BucheonCityHall|BucheonCityHall]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]
* 07:14 [[phab:p/BucheonFesta|BucheonFesta]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2026-03-08 ===
* 14:34 [[phab:p/Unicord|Unicord]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]

=== 2026-02-22 ===
* 10:39 [[phab:p/Kredionecsresmi|Kredionecsresmi]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2026-02-15 ===
* 03:34 [[phab:p/alfredbeck|alfredbeck]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2026-02-11 ===
* 14:34 [[phab:p/Nguyentrongphu|Nguyentrongphu]] was disabled by [[phab:p/Superpes15/|Superpes15]]
* 14:32 [[phab:p/Slowking4|Slowking4]] was disabled by [[phab:p/Superpes15/|Superpes15]]

=== 2026-02-09 ===
* 22:45 [[phab:p/UNIX-QUANTUM-UNIBANK-FICSIT-NETWORKS|UNIX-QUANTUM-UNIBANK-FICSIT-NETWORKS]] was disabled by [[phab:p/Zabe/|Zabe]]

=== 2026-01-23 ===
* 12:09 [[phab:p/Aboodhassanio|Aboodhassanio]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2026-01-16 ===
* 16:08 [[phab:p/Kimlien316|Kimlien316]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]
* 12:01 [[phab:p/Batiste67400|Batiste67400]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2026-01-12 ===
* 22:04 [[phab:p/Claudioluna6|Claudioluna6]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2026-01-03 ===
* 14:31 [[phab:p/Cw95hh9|Cw95hh9]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2025-12-27 ===
* 06:28 [[phab:p/LBLaiSiNanHai|LBLaiSiNanHai]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-12-24 ===
* 21:36 [[phab:p/ItsLido|ItsLido]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-12-11 ===
* 15:50 [[phab:p/DanielJohnso|DanielJohnso]] was disabled by [[phab:p/Marostegui/|Marostegui]]

=== 2025-12-01 ===
* 20:36 [[phab:p/Lkcl|Lkcl]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2025-11-30 ===
* 03:56 [[phab:p/BscottAPL33|BscottAPL33]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-11-28 ===
* 21:21 [[phab:p/3894djfj.10439djf|3894djfj.10439djf]] was disabled by [[phab:p/JJMC89/|JJMC89]]
* 09:35 [[phab:p/ElinagittHuB|ElinagittHuB]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-11-22 ===
* 22:09 [[phab:p/Aadvertising|Aadvertising]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 22:09 [[phab:p/TweakFind|TweakFind]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-11-20 ===
* 04:46 [[phab:p/SydneyRug|SydneyRug]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-11-14 ===
* 08:52 [[phab:p/Lougrammfoundation|Lougrammfoundation]] was disabled by [[phab:p/Marostegui/|Marostegui]]

=== 2025-11-12 ===
* 06:59 [[phab:p/Cesar|Cesar]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-11-05 ===
* 23:03 [[phab:p/Nehtechnine|Nehtechnine]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-11-02 ===
* 06:33 [[phab:p/Chhoundevid|Chhoundevid]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2025-11-01 ===
* 07:11 [[phab:p/TowfiqSir|TowfiqSir]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2025-10-14 ===
* 12:30 [[phab:p/Amrok84|Amrok84]] was disabled by [[phab:p/Lucas_Werkmeister_WMDE/|Lucas_Werkmeister_WMDE]]

=== 2025-09-24 ===
* 13:31 [[phab:p/100592|100592]] was disabled by [[phab:p/A_smart_kitten/|A_smart_kitten]]

=== 2025-09-22 ===
* 16:20 [[phab:p/DanishAhmedKm|DanishAhmedKm]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-09-12 ===
* 14:08 [[phab:p/GrimGwTK|GrimGwTK]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-09-04 ===
* 15:14 [[phab:p/Amarvip|Amarvip]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-08-29 ===
* 00:09 [[phab:p/IbJo|IbJo]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-08-27 ===
* 02:31 [[phab:p/W11228|W11228]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-08-25 ===
* 17:34 [[phab:p/Faster_than_Thunder|Faster_than_Thunder]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-08-22 ===
* 22:07 [[phab:p/T200856-01|T200856-01]] was disabled by [[phab:p/bd808/|bd808]]

=== 2025-08-19 ===
* 04:58 [[phab:p/tiffatk|tiffatk]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2025-08-15 ===
* 05:00 [[phab:p/Totrue89|Totrue89]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-08-06 ===
* 22:20 [[phab:p/Sajidali110|Sajidali110]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-07-31 ===
* 10:17 [[phab:p/EMIZENTECH|EMIZENTECH]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-07-30 ===
* 10:21 [[phab:p/Fariha_Asghar785|Fariha_Asghar785]] was disabled by [[phab:p/Lucas_Werkmeister_WMDE/|Lucas_Werkmeister_WMDE]]

=== 2025-07-08 ===
* 10:46 [[phab:p/Tulsi_Bhagat|Tulsi_Bhagat]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2025-06-29 ===
* 16:12 [[phab:p/sassybritches|sassybritches]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-06-22 ===
* 07:54 [[phab:p/Godspowertechnical|Godspowertechnical]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-06-08 ===
* 14:59 [[phab:p/Jaypam001|Jaypam001]] was disabled by [[phab:p/JJMC89/|JJMC89]]
* 14:53 [[phab:p/DANISHAHMED111|DANISHAHMED111]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-06-07 ===
* 05:50 [[phab:p/PCJND|PCJND]] was disabled by [[phab:p/Johannnes89/|Johannnes89]]

=== 2025-06-04 ===
* 08:37 [[phab:p/Alpasli|Alpasli]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2025-06-03 ===
* 02:07 [[phab:p/Jj881|Jj881]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-05-29 ===
* 05:51 [[phab:p/RodneyAraujo|RodneyAraujo]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2025-04-28 ===
* 15:07 [[phab:p/Hansmuller|Hansmuller]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2025-04-03 ===
* 15:57 [[phab:p/Wfan|Wfan]] was disabled by [[phab:p/Zabe/|Zabe]]

=== 2025-03-30 ===
* 10:15 [[phab:p/Watnoii24|Watnoii24]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-03-23 ===
* 11:27 [[phab:p/Saadtbli|Saadtbli]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-03-22 ===
* 16:45 [[phab:p/Stephonjeffries19|Stephonjeffries19]] was disabled by [[phab:p/LucasWerkmeister/|LucasWerkmeister]]
* 04:32 [[phab:p/Chriswarriortv|Chriswarriortv]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-03-19 ===
* 11:29 [[phab:p/Vinay080|Vinay080]] was disabled by [[phab:p/zeljkofilipin/|zeljkofilipin]]

=== 2025-03-18 ===
* 12:04 [[phab:p/Walshandpartners777|Walshandpartners777]] was disabled by [[phab:p/Lucas_Werkmeister_WMDE/|Lucas_Werkmeister_WMDE]]

=== 2025-03-04 ===
* 01:33 [[phab:p/Porokhov|Porokhov]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2025-02-25 ===
* 17:27 [[phab:p/Selahaddin751|Selahaddin751]] was disabled by [[phab:p/brennen/|brennen]]

=== 2025-02-19 ===
* 01:00 [[phab:p/Mrb_Rafi|Mrb_Rafi]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2025-02-14 ===
* 19:19 [[phab:p/3652candy|3652candy]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 17:01 [[phab:p/Ataysaa|Ataysaa]] was disabled by [[phab:p/bd808/|bd808]]

=== 2025-02-09 ===
* 09:10 [[phab:p/BTullis|BTullis]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2025-02-08 ===
* 23:26 [[phab:p/Alexdivkovic05|Alexdivkovic05]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-02-06 ===
* 06:19 [[phab:p/HormigasAIS|HormigasAIS]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-01-29 ===
* 07:40 [[phab:p/Denker61|Denker61]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2025-01-25 ===
* 21:36 [[phab:p/Khnthichith|Khnthichith]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-01-24 ===
* 11:33 [[phab:p/Aek191010|Aek191010]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2025-01-05 ===
* 15:40 [[phab:p/szsuperzuper|szsuperzuper]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2025-01-01 ===
* 09:08 [[phab:p/GALAXYENTERPRISES|GALAXYENTERPRISES]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-12-20 ===
* 00:52 [[phab:p/Mail.faluzes|Mail.faluzes]] was disabled by [[phab:p/Reedy/|Reedy]]

=== 2024-12-13 ===
* 02:01 [[phab:p/Gussdafii|Gussdafii]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-12-11 ===
* 08:54 [[phab:p/CodeTrailblazer|CodeTrailblazer]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 08:54 [[phab:p/SelvikIN|SelvikIN]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-12-03 ===
* 05:16 [[phab:p/Matkospajdr|Matkospajdr]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-12-01 ===
* 19:27 [[phab:p/Adarshsingh|Adarshsingh]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-11-28 ===
* 22:47 [[phab:p/Sandraklemma|Sandraklemma]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-11-23 ===
* 09:00 [[phab:p/Mahimabajpayee12|Mahimabajpayee12]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-11-11 ===
* 11:09 [[phab:p/Mvwservices|Mvwservices]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 07:00 [[phab:p/Impactolog|Impactolog]] was disabled by [[phab:p/revi/|revi]]

=== 2024-10-30 ===
* 09:05 [[phab:p/Jweighed1|Jweighed1]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-10-25 ===
* 04:20 [[phab:p/Blunt2531|Blunt2531]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-10-08 ===
* 08:31 [[phab:p/Surfcityrecovery|Surfcityrecovery]] was disabled by [[phab:p/MoritzMuehlenhoff/|MoritzMuehlenhoff]]

=== 2024-10-01 ===
* 21:49 [[phab:p/T200856-01|T200856-01]] was disabled by [[phab:p/bd808/|bd808]]

=== 2024-09-27 ===
* 10:20 [[phab:p/SorBP|SorBP]] was disabled by [[phab:p/TheresNoTime/|TheresNoTime]]

=== 2024-09-08 ===
* 10:45 [[phab:p/Robin_Mathew_Rajan|Robin_Mathew_Rajan]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2024-09-02 ===
* 17:58 [[phab:p/Idxntcx|Idxntcx]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-09-01 ===
* 10:11 [[phab:p/LDAP|LDAP]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-07-22 ===
* 08:56 [[phab:p/Nobleadele|Nobleadele]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-06-18 ===
* 20:54 [[phab:p/Playgiirlkaybrazy|Playgiirlkaybrazy]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-06-08 ===
* 22:30 [[phab:p/Exposingsesion1|Exposingsesion1]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-05-27 ===
* 12:24 [[phab:p/JosefineHellrothLarssonWMSE|JosefineHellrothLarssonWMSE]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 07:54 [[phab:p/SMMpanels|SMMpanels]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-05-22 ===
* 11:31 [[phab:p/Sandra_Fauconnier_WMSE|Sandra_Fauconnier_WMSE]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:23 [[phab:p/MiaJacobssonWMSE|MiaJacobssonWMSE]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:20 [[phab:p/David_Haskiya_WMSE|David_Haskiya_WMSE]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:20 [[phab:p/kalle|kalle]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:20 [[phab:p/Tore_Danielsson_WMSE|Tore_Danielsson_WMSE]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:19 [[phab:p/Gitta|Gitta]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:19 [[phab:p/annatroberg|annatroberg]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:19 [[phab:p/AxelPettersson_WMSE|AxelPettersson_WMSE]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]
* 10:17 [[phab:p/SaraMortsell|SaraMortsell]] was disabled by [[phab:p/Sebastian_Berlin-WMSE/|Sebastian_Berlin-WMSE]]

=== 2024-05-13 ===
* 14:55 [[phab:p/BenoitPrieur|BenoitPrieur]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2024-05-04 ===
* 19:54 [[phab:p/Sammoon391|Sammoon391]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-05-01 ===
* 07:24 [[phab:p/Soubag|Soubag]] was disabled by [[phab:p/Mainframe98/|Mainframe98]]

=== 2024-04-28 ===
* 06:22 [[phab:p/Diamondscoin|Diamondscoin]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-04-19 ===
* 03:47 [[phab:p/Wawmart2|Wawmart2]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-04-08 ===
* 10:21 [[phab:p/Mardetanha|Mardetanha]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2024-03-29 ===
* 09:03 [[phab:p/Abdollmjjedloveanan|Abdollmjjedloveanan]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-03-12 ===
* 09:45 [[phab:p/Samantha78462|Samantha78462]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 09:39 [[phab:p/Samantha7861654654|Samantha7861654654]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 09:11 [[phab:p/Robin|Robin]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 09:05 [[phab:p/Anglinakuki|Anglinakuki]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 00:02 [[phab:p/Johnne25|Johnne25]] was disabled by [[phab:p/bd808/|bd808]]

=== 2024-03-07 ===
* 20:07 [[phab:p/Sami785|Sami785]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-03-06 ===
* 07:41 [[phab:p/28q|28q]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2024-03-02 ===
* 22:51 [[phab:p/kitchenstrategic|kitchenstrategic]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-02-23 ===
* 09:03 [[phab:p/littleggghost|littleggghost]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-02-17 ===
* 05:10 [[phab:p/Skekeiei|Skekeiei]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-01-27 ===
* 03:50 [[phab:p/Andybitcoin|Andybitcoin]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2024-01-25 ===
* 09:10 [[phab:p/Mayo3030|Mayo3030]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-01-21 ===
* 05:59 [[phab:p/Hackear|Hackear]] was disabled by [[phab:p/Peachey88/|Peachey88]]
* 05:58 [[phab:p/joselopez45|joselopez45]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-01-20 ===
* 20:02 [[phab:p/08107130655|08107130655]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2024-01-18 ===
* 05:33 [[phab:p/Tecnologynew|Tecnologynew]] was disabled by [[phab:p/TheresNoTime/|TheresNoTime]]

=== 2024-01-12 ===
* 22:34 [[phab:p/cchen|cchen]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2024-01-11 ===
* 07:34 [[phab:p/Bernita43|Bernita43]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2024-01-07 ===
* 10:13 [[phab:p/Irademack|Irademack]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2023-12-31 ===
* 16:48 [[phab:p/Vieclamdmpt|Vieclamdmpt]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2023-12-26 ===
* 20:34 [[phab:p/Bgu5678|Bgu5678]] was disabled by [[phab:p/Peachey88/|Peachey88]]

=== 2023-11-26 ===
* 20:09 [[phab:p/Str13tlife|Str13tlife]] was disabled by [[phab:p/JJMC89/|JJMC89]]

=== 2023-11-24 ===
* 00:49 [[phab:p/Imambuchori03|Imambuchori03]] was disabled by [[phab:p/DannyS712/|DannyS712]]

=== 2023-11-22 ===
* 15:50 [[phab:p/Naleksuh|Naleksuh]] was disabled by [[phab:p/WMFOffice/|WMFOffice]]

=== 2023-11-19 ===
* 21:10 [[phab:p/Onack16888|Onack16888]] was disabled by [[phab:p/Daimona/|Daimona]]

=== 2023-11-15 ===
* 11:57 [[phab:p/Anonymous_ehacker|Anonymous_ehacker]] was disabled by [[phab:p/hashar/|hashar]]

=== 2023-11-09 ===
* 23:16 [[phab:p/dunicorn|dunicorn]] was disabled by [[phab:p/bd808/|bd808]]

=== 2023-09-30 ===
* 17:38 [[phab:p/Wykirany|Wykirany]] was disabled by [[phab:p/RhinosF1/|RhinosF1]]

=== 2023-09-01 ===
* 16:29 [[phab:p/T200856-01|T200856-01]] was disabled by [[phab:p/bd808/|bd808]]
* 16:17 [[phab:p/T200856-01|T200856-01]] was disabled by [[phab:p/bd808/|bd808]]</text>
      <sha1>rtzxgz1lseolqnuka47uknfnu0xuapj</sha1>
    </revision>
  </page>
  <page>
    <title>Data Platform/Systems/Airflow/Kubernetes/Operations</title>
    <ns>0</ns>
    <id>456157</id>
    <revision>
      <id>2414327</id>
      <parentid>2396278</parentid>
      <timestamp>2026-05-16T22:16:24Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>fix &quot;here&quot; as link label; fix broken links</comment>
      <origin>2414327</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="10698" sha1="t4jxd9fjdfve1v1d99i2taolddfhvpg" xml:space="preserve">
=== Creating a new instance ===
{{Note|text=In the following section, we're going to assume that we're creating a new airflow instance named &lt;code&gt;airflow-example&lt;/code&gt;, deployed with a dedicated PG cluster named &lt;code&gt;postgresql-airflow-example&lt;/code&gt;, deployed in the &lt;code&gt;dse-k8s-eqiad&lt;/code&gt; Kubernetes environment, accessible via https://airflow-example.wikimedia.org}}

* The first thing you need to do is create Kubernetes [[gerrit:c/operations/puppet/+/1043277/4/hieradata/common/profile/kubernetes/deployment_server.yaml|read and deploy user credentials]]
* Add a namespace (using the same name as the airflow instance) entry into &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s.yaml&lt;/code&gt;&lt;syntaxhighlight lang=&quot;yaml&quot;&gt;
namespaces:
  # ...
  airflow-example:
    deployClusterRole: deploy-airflow
    customPriorityClasses: [pod-critical]
    tlsExtraSANs:
      - airflow-example.wikimedia.org
&lt;/syntaxhighlight&gt;
* Add the namespace under the &lt;code&gt;tenantNamespaces&lt;/code&gt; list in &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s-eqiad/cephfs-csi-rbd-values.yaml&lt;/code&gt; as well as &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s-eqiad/cephfs-csi-rbd-values.yaml&lt;/code&gt;
* Add the namespace to the &lt;code&gt;watchedNamespaces&lt;/code&gt; list defined in &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s-eqiad/cloudnative-pg.yaml&lt;/code&gt;
* Then, create the [[gerrit:c/operations/dns/+/1062048|public and internal DNS records]] for this instance
* Define the PG cluster and airflow instance &lt;code&gt;helmfile.yaml&lt;/code&gt; files and associated values (take example from &lt;code&gt;deployment_charts/helmfile.d/dse-k8s-services/postgresql-airflow-example&lt;/code&gt; and &lt;code&gt;deployment_charts/helmfile.d/dse-k8s-services/airflow-example&lt;/code&gt;)
* Generate the S3 keypairs for both PG and Airflow&lt;syntaxhighlight lang=&quot;bash&quot;&gt;brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=postgresql-airflow-example --display-name=&quot;postgresql-airflow-example&quot;
brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=airflow-example --display-name=&quot;airflow-example&quot;
# note: copy the `access_key` and `secret_key` from the JSON output, you will need it in the next step&lt;/syntaxhighlight&gt;
* Create the S3 buckets for both PG and Airflow&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@stat1008:~$ read access_key
&lt;PG S3 access key&gt;
brouberol@stat1008:~$ read secret_key
&lt;PG S3 secret key&gt;
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://postgresql-airflow-example.dse-k8s-eqiad
brouberol@stat1008:~$ read access_key
&lt;Airflow S3 access key&gt;
brouberol@stat1008:~$ read secret_key
&lt;Airflow S3 secret key&gt;
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://logs.airflow-example.dse-k8s-eqiad
&lt;/syntaxhighlight&gt;

* [[gerrit:c/operations/puppet/+/1057799|Register the service in our IDP server]] (into &lt;code&gt;idp.yaml&lt;/code&gt;). After the patch was merged and puppet ran on the idp servers, copy the OIDC secret key generated for the airflow service.&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@idp1004:# cat /etc/cas/services/airflow_example-*.json  | jq -r .clientSecret
&lt;OIDC secret key&gt;
&lt;/syntaxhighlight&gt;
* Issue a Kerberos keytab following the guide provided on [[Data Platform/Systems/Kerberos/Administration]]. The hostname must match the internal service discovery domain of the airflow instance.&lt;syntaxhighlight lang=&quot;bash&quot;&gt;# Change `analytics-example` to the UNIX user the airflow tasks will impersonate by default in Hadoop
brouberol@krb1002:~$ sudo kadmin.local addprinc -randkey analytics-example/airflow-example.discovery.wmnet@WIKIMEDIA
brouberol@krb1002:~$ sudo kadmin.local addprinc -randkey airflow/airflow-example.discovery.wmnet@WIKIMEDIA
brouberol@krb1002:~$ sudo kadmin.local addprinc -randkey HTTP/airflow-example.discovery.wmnet@WIKIMEDIA
brouberol@krb1002:~$ sudo kadmin.local ktadd -norandkey -k analytics-example.keytab \
    analytics-example/airflow-example.discovery.wmnet \
    airflow/airflow-example.discovery.wmnet@WIKIMEDIA \
    HTTP/airflow-example.discovery.wmnet@WIKIMEDIA&lt;/syntaxhighlight&gt;

Created the base64 representation for the keytab
 root@krb1002:~# base64 analytics-example.keytab

* Generate the secrets or both the PG cluster and the Airflow instance and add the to the private puppet repository, to &lt;code&gt;/srv/git/private/hieradata/role/common/deployment_server/kubernetes.yaml&lt;/code&gt;&lt;syntaxhighlight lang=&quot;yaml&quot;&gt;dse-k8s:
  # ...
  postgresql-airflow-example:
    dse-k8s-eqiad:
      s3:
        accessKey: &lt;PG S3 access key&gt;
        secretKey: &lt;PG S3 secret key&gt;

  airflow-example:
    dse-k8s-eqiad:
      config:
        private:
          airflow__core__fernet_key: &lt;random 64 characters&gt;
          airflow__webserver__secret_key: &lt;random 64 characters&gt;
        airflow:
          aws_access_key_id: &lt;Airflow S3 access key&gt;
          aws_secret_access_key: &lt;Airflow S3 secret key&gt;
        oidc:
          client_secret: &lt;OIDC secret key&gt;
      kerberos:
        keytab: | # The base64 representation obtained in the previous step
          ABCDEFHIJKLMNOP
          ABCDEFHIJKLMNOP=&lt;/syntaxhighlight&gt;
* Register the PG bucket name and keys into &lt;code&gt;/srv/git/private/hieradata/role/common/mariadb/misc/analytics/backup.yaml&lt;/code&gt; on the &lt;code&gt;puppetserver&lt;/code&gt; host. This will make sure that the PG base backups and WALs are regularly backed up outside of our Ceph cluster.&lt;syntaxhighlight lang=&quot;yaml&quot;&gt;
profile::ceph::backup::s3_local::sources:
  ...
  postgresql-airflow-example.dse-k8s-eqiad: # must match the PG bucket name 
    access_key: &lt;PG S3 access key&gt;
    secret_key: &lt;PG S3 secret key&gt;
&lt;/syntaxhighlight&gt;
* [[Kubernetes/Deployments#Deploying with helmfile|Deploy]] the service (which should deploy both the PG cluster and the airflow instance)
* Once the instance is running, [[gerrit:c/operations/puppet/+/1063848|enable]] the ATS redirection from the wikimedia.org subdomain to the kube ingress. After puppet has run on all the cache servers (wait a good 30 minutes), [https://airflow-example.wikimedia.org/ https://airflow-example.wikimedia.org] should display the airflow web UI, and you should be able to connect via CAS.

==== Hadoop setup ====
Once the instance is up and running, we need to make sure Hadoop is correctly configured to support it. 

'''Note''': These instructions will assume that the airflow instance was deployed using a new Kerberos principal primary (that we'll name &lt;code&gt;analytics-example&lt;/code&gt; in that section). If you're reusing an existing principal, such as &lt;code&gt;analytics&lt;/code&gt;, you can stop there.

===== Create UNIX user/group =====

* Define an analytics-example user and group in [[gerrit:plugins/gitiles/operations/puppet/+/73271677c2c06190e575a33f039bbcf90b8802ec/modules/admin/data/data.yaml#1036|&lt;code&gt;data.yaml&lt;/code&gt;]] and add it to the &lt;code&gt;analytics-privatedata-users&lt;/code&gt; group.
* In the same file, define an &lt;code&gt;analytics-example-users&lt;/code&gt; group containing all human users requiring to impersonate the primary

===== Create the HDFS folders =====

* Create a home directory in HDFS for that user&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -mkdir /user/analytics-example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown analytics-example /user/analytics-example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chgrp analytics-example-users /user/analytics-example
&lt;/syntaxhighlight&gt;
* Create a temporary directory for the airflow instance&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -mkdir /tmp/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown analytics-example /tmp/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown analytics-privatedata-users /tmp/example
&lt;/syntaxhighlight&gt;
* Create an artifact directory for the airflow instance&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -mkdir /wmf/cache/artifacts/airflow/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown blunderbuss /wmf/cache/artifacts/airflow/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chgrp blunderbuss /wmf/cache/artifacts/airflow/example
&lt;/syntaxhighlight&gt;

===== Configure the new instance in [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags airflow-dags] repo =====

* Add the new &lt;code&gt;analytics-example&lt;/code&gt; folder with new &lt;code&gt;dags&lt;/code&gt; and &lt;code&gt;config&lt;/code&gt; subfolders
* Add new &lt;code&gt;config/dag_config.py&lt;/code&gt; and &lt;code&gt;config/artifacts.yaml&lt;/code&gt; files specific to the &lt;code&gt;analytics-example&lt;/code&gt; instance
* Add the correct tests folder &lt;code&gt;tests/analytics-example&lt;/code&gt;
* Update [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/.gitlab-ci.yml%7C&lt;code&gt;.gitlab-ci.yml&lt;/code&gt;] and add the new instance's name to the Blunderbuss cache updater's instance lists at [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/.gitlab-ci.yml?ref_type=heads#L7 location one] and [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/.gitlab-ci.yml?ref_type=heads#L168 location two]

=== Configuring out-of-band backups ===
The PostgreSQL database cluster for this instance will already be configured with its own backup system that writes database backups and WAL archives to [[Data Platform/Systems/Ceph#Object Storage|the S3 interface of the Ceph cluster]].

However, we decided to implement out-of-band backups of each of the S3 buckets containing these database backups, so we added a new backup pipeline to [[Data Platform/Systems/DB Replica|our database backup replica]] system, which is '''db1208'''.

In this case the file you need to modify when you add a new instance is in the '''private''' repo and is named: &lt;code&gt;hieradata/role/common/mariadb/misc/analytics/backup.yaml&lt;/code&gt;

Add your new bucket and its access credentials to the &lt;code&gt;profile::ceph::backup::s3_local::sources&lt;/code&gt; hash structure, as shown.&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
profile::ceph::backup::s3_local::sources:
  postgresql.airflow-example-k8s-eqiad:
    access_key: &lt;Airflow S3 access key&gt;
    secret_key: &lt;Airflow S3 secret key&gt;
&lt;/syntaxhighlight&gt;When merged, this will update the file &lt;code&gt;/srv/postgresql_backups/rclone.conf&lt;/code&gt; on db1208, adding the backups of this database cluster to the daily sync process and therefore to [[Bacula]].

=== Kubernetes migration operations ===
[[Data Platform/Systems/Airflow/Kubernetes/Operations/K8s-Migration]]</text>
      <sha1>t4jxd9fjdfve1v1d99i2taolddfhvpg</sha1>
    </revision>
    <revision>
      <id>2414329</id>
      <parentid>2414327</parentid>
      <timestamp>2026-05-16T22:23:42Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix [[w:WP:LISTGAP]]</comment>
      <origin>2414329</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="10697" sha1="s8mhxx1sau6rq2n9hjouhe0wyj3dok4" xml:space="preserve">
=== Creating a new instance ===
{{Note|text=In the following section, we're going to assume that we're creating a new airflow instance named &lt;code&gt;airflow-example&lt;/code&gt;, deployed with a dedicated PG cluster named &lt;code&gt;postgresql-airflow-example&lt;/code&gt;, deployed in the &lt;code&gt;dse-k8s-eqiad&lt;/code&gt; Kubernetes environment, accessible via https://airflow-example.wikimedia.org}}

* The first thing you need to do is create Kubernetes [[gerrit:c/operations/puppet/+/1043277/4/hieradata/common/profile/kubernetes/deployment_server.yaml|read and deploy user credentials]]
* Add a namespace (using the same name as the airflow instance) entry into &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s.yaml&lt;/code&gt;&lt;syntaxhighlight lang=&quot;yaml&quot;&gt;
namespaces:
  # ...
  airflow-example:
    deployClusterRole: deploy-airflow
    customPriorityClasses: [pod-critical]
    tlsExtraSANs:
      - airflow-example.wikimedia.org
&lt;/syntaxhighlight&gt;
* Add the namespace under the &lt;code&gt;tenantNamespaces&lt;/code&gt; list in &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s-eqiad/cephfs-csi-rbd-values.yaml&lt;/code&gt; as well as &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s-eqiad/cephfs-csi-rbd-values.yaml&lt;/code&gt;
* Add the namespace to the &lt;code&gt;watchedNamespaces&lt;/code&gt; list defined in &lt;code&gt;deployment_charts/helmfile.d/admin_ng/values/dse-k8s-eqiad/cloudnative-pg.yaml&lt;/code&gt;
* Then, create the [[gerrit:c/operations/dns/+/1062048|public and internal DNS records]] for this instance
* Define the PG cluster and airflow instance &lt;code&gt;helmfile.yaml&lt;/code&gt; files and associated values (take example from &lt;code&gt;deployment_charts/helmfile.d/dse-k8s-services/postgresql-airflow-example&lt;/code&gt; and &lt;code&gt;deployment_charts/helmfile.d/dse-k8s-services/airflow-example&lt;/code&gt;)
* Generate the S3 keypairs for both PG and Airflow&lt;syntaxhighlight lang=&quot;bash&quot;&gt;brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=postgresql-airflow-example --display-name=&quot;postgresql-airflow-example&quot;
brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=airflow-example --display-name=&quot;airflow-example&quot;
# note: copy the `access_key` and `secret_key` from the JSON output, you will need it in the next step&lt;/syntaxhighlight&gt;
* Create the S3 buckets for both PG and Airflow&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@stat1008:~$ read access_key
&lt;PG S3 access key&gt;
brouberol@stat1008:~$ read secret_key
&lt;PG S3 secret key&gt;
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://postgresql-airflow-example.dse-k8s-eqiad
brouberol@stat1008:~$ read access_key
&lt;Airflow S3 access key&gt;
brouberol@stat1008:~$ read secret_key
&lt;Airflow S3 secret key&gt;
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://logs.airflow-example.dse-k8s-eqiad
&lt;/syntaxhighlight&gt;
* [[gerrit:c/operations/puppet/+/1057799|Register the service in our IDP server]] (into &lt;code&gt;idp.yaml&lt;/code&gt;). After the patch was merged and puppet ran on the idp servers, copy the OIDC secret key generated for the airflow service.&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@idp1004:# cat /etc/cas/services/airflow_example-*.json  | jq -r .clientSecret
&lt;OIDC secret key&gt;
&lt;/syntaxhighlight&gt;
* Issue a Kerberos keytab following the guide provided on [[Data Platform/Systems/Kerberos/Administration]]. The hostname must match the internal service discovery domain of the airflow instance.&lt;syntaxhighlight lang=&quot;bash&quot;&gt;# Change `analytics-example` to the UNIX user the airflow tasks will impersonate by default in Hadoop
brouberol@krb1002:~$ sudo kadmin.local addprinc -randkey analytics-example/airflow-example.discovery.wmnet@WIKIMEDIA
brouberol@krb1002:~$ sudo kadmin.local addprinc -randkey airflow/airflow-example.discovery.wmnet@WIKIMEDIA
brouberol@krb1002:~$ sudo kadmin.local addprinc -randkey HTTP/airflow-example.discovery.wmnet@WIKIMEDIA
brouberol@krb1002:~$ sudo kadmin.local ktadd -norandkey -k analytics-example.keytab \
    analytics-example/airflow-example.discovery.wmnet \
    airflow/airflow-example.discovery.wmnet@WIKIMEDIA \
    HTTP/airflow-example.discovery.wmnet@WIKIMEDIA&lt;/syntaxhighlight&gt;

Created the base64 representation for the keytab
 root@krb1002:~# base64 analytics-example.keytab

* Generate the secrets or both the PG cluster and the Airflow instance and add the to the private puppet repository, to &lt;code&gt;/srv/git/private/hieradata/role/common/deployment_server/kubernetes.yaml&lt;/code&gt;&lt;syntaxhighlight lang=&quot;yaml&quot;&gt;dse-k8s:
  # ...
  postgresql-airflow-example:
    dse-k8s-eqiad:
      s3:
        accessKey: &lt;PG S3 access key&gt;
        secretKey: &lt;PG S3 secret key&gt;

  airflow-example:
    dse-k8s-eqiad:
      config:
        private:
          airflow__core__fernet_key: &lt;random 64 characters&gt;
          airflow__webserver__secret_key: &lt;random 64 characters&gt;
        airflow:
          aws_access_key_id: &lt;Airflow S3 access key&gt;
          aws_secret_access_key: &lt;Airflow S3 secret key&gt;
        oidc:
          client_secret: &lt;OIDC secret key&gt;
      kerberos:
        keytab: | # The base64 representation obtained in the previous step
          ABCDEFHIJKLMNOP
          ABCDEFHIJKLMNOP=&lt;/syntaxhighlight&gt;
* Register the PG bucket name and keys into &lt;code&gt;/srv/git/private/hieradata/role/common/mariadb/misc/analytics/backup.yaml&lt;/code&gt; on the &lt;code&gt;puppetserver&lt;/code&gt; host. This will make sure that the PG base backups and WALs are regularly backed up outside of our Ceph cluster.&lt;syntaxhighlight lang=&quot;yaml&quot;&gt;
profile::ceph::backup::s3_local::sources:
  ...
  postgresql-airflow-example.dse-k8s-eqiad: # must match the PG bucket name 
    access_key: &lt;PG S3 access key&gt;
    secret_key: &lt;PG S3 secret key&gt;
&lt;/syntaxhighlight&gt;
* [[Kubernetes/Deployments#Deploying with helmfile|Deploy]] the service (which should deploy both the PG cluster and the airflow instance)
* Once the instance is running, [[gerrit:c/operations/puppet/+/1063848|enable]] the ATS redirection from the wikimedia.org subdomain to the kube ingress. After puppet has run on all the cache servers (wait a good 30 minutes), [https://airflow-example.wikimedia.org/ https://airflow-example.wikimedia.org] should display the airflow web UI, and you should be able to connect via CAS.

==== Hadoop setup ====
Once the instance is up and running, we need to make sure Hadoop is correctly configured to support it. 

'''Note''': These instructions will assume that the airflow instance was deployed using a new Kerberos principal primary (that we'll name &lt;code&gt;analytics-example&lt;/code&gt; in that section). If you're reusing an existing principal, such as &lt;code&gt;analytics&lt;/code&gt;, you can stop there.

===== Create UNIX user/group =====

* Define an analytics-example user and group in [[gerrit:plugins/gitiles/operations/puppet/+/73271677c2c06190e575a33f039bbcf90b8802ec/modules/admin/data/data.yaml#1036|&lt;code&gt;data.yaml&lt;/code&gt;]] and add it to the &lt;code&gt;analytics-privatedata-users&lt;/code&gt; group.
* In the same file, define an &lt;code&gt;analytics-example-users&lt;/code&gt; group containing all human users requiring to impersonate the primary

===== Create the HDFS folders =====

* Create a home directory in HDFS for that user&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -mkdir /user/analytics-example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown analytics-example /user/analytics-example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chgrp analytics-example-users /user/analytics-example
&lt;/syntaxhighlight&gt;
* Create a temporary directory for the airflow instance&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -mkdir /tmp/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown analytics-example /tmp/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown analytics-privatedata-users /tmp/example
&lt;/syntaxhighlight&gt;
* Create an artifact directory for the airflow instance&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -mkdir /wmf/cache/artifacts/airflow/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chown blunderbuss /wmf/cache/artifacts/airflow/example
brouberol@an-master1003:~$ sudo -u hdfs /usr/bin/hdfs dfs -chgrp blunderbuss /wmf/cache/artifacts/airflow/example
&lt;/syntaxhighlight&gt;

===== Configure the new instance in [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags airflow-dags] repo =====

* Add the new &lt;code&gt;analytics-example&lt;/code&gt; folder with new &lt;code&gt;dags&lt;/code&gt; and &lt;code&gt;config&lt;/code&gt; subfolders
* Add new &lt;code&gt;config/dag_config.py&lt;/code&gt; and &lt;code&gt;config/artifacts.yaml&lt;/code&gt; files specific to the &lt;code&gt;analytics-example&lt;/code&gt; instance
* Add the correct tests folder &lt;code&gt;tests/analytics-example&lt;/code&gt;
* Update [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/.gitlab-ci.yml%7C&lt;code&gt;.gitlab-ci.yml&lt;/code&gt;] and add the new instance's name to the Blunderbuss cache updater's instance lists at [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/.gitlab-ci.yml?ref_type=heads#L7 location one] and [https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/.gitlab-ci.yml?ref_type=heads#L168 location two]

=== Configuring out-of-band backups ===
The PostgreSQL database cluster for this instance will already be configured with its own backup system that writes database backups and WAL archives to [[Data Platform/Systems/Ceph#Object Storage|the S3 interface of the Ceph cluster]].

However, we decided to implement out-of-band backups of each of the S3 buckets containing these database backups, so we added a new backup pipeline to [[Data Platform/Systems/DB Replica|our database backup replica]] system, which is '''db1208'''.

In this case the file you need to modify when you add a new instance is in the '''private''' repo and is named: &lt;code&gt;hieradata/role/common/mariadb/misc/analytics/backup.yaml&lt;/code&gt;

Add your new bucket and its access credentials to the &lt;code&gt;profile::ceph::backup::s3_local::sources&lt;/code&gt; hash structure, as shown.&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
profile::ceph::backup::s3_local::sources:
  postgresql.airflow-example-k8s-eqiad:
    access_key: &lt;Airflow S3 access key&gt;
    secret_key: &lt;Airflow S3 secret key&gt;
&lt;/syntaxhighlight&gt;When merged, this will update the file &lt;code&gt;/srv/postgresql_backups/rclone.conf&lt;/code&gt; on db1208, adding the backups of this database cluster to the daily sync process and therefore to [[Bacula]].

=== Kubernetes migration operations ===
[[Data Platform/Systems/Airflow/Kubernetes/Operations/K8s-Migration]]</text>
      <sha1>s8mhxx1sau6rq2n9hjouhe0wyj3dok4</sha1>
    </revision>
  </page>
  <page>
    <title>Portal:Cloud VPS/Admin/Magnum setup</title>
    <ns>0</ns>
    <id>459115</id>
    <revision>
      <id>2414322</id>
      <parentid>2411268</parentid>
      <timestamp>2026-05-16T22:03:56Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <minor />
      <comment>fix &quot;here&quot; as link label</comment>
      <origin>2414322</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="12482" sha1="1iwfro39os01qdx2wfytxd3qujouvbx" xml:space="preserve">= Intro =

[http://Portal:Cloud%20VPS/Admin/Magnum Magnum] relies on a host of moving parts to launch a Kubernetes cluster. As of 2025, there are two back-end implementations. For the last few years we've been using the [https://wiki.openstack.org/wiki/Heat Heat] backend, but it is being deprecated by the upstream developers in favor of cluster-api backends.

We are in the process of moving from the Heat backend to the [https://docs.openstack.org/magnum-capi-helm/latest/ capi helm] backend. This document is about understanding and setting up Magnum to work with capi helm drivers.

= Openstack Components =

When creating a cluster, Magnum creates a service account and credentials in Keystone and then delegates the actual cluster creation to the cluster-api service which uses the Magnum-managed credentials. Cluster-api uses those credentials to call back into other openstack services to create and configure the components of the cluster (e.g. calling nova to create a worker node) and then applies helm charts to the nodes to set up the moving parts.

== Magnum-api ==
Magnum-api provides the REST api for managing kubernetes clusters and cluster templates. It is a straightforward python/uwsgi service that runs on cloudcontrol nodes and is managed by puppet just like other openstack services.

The magnum-api logs can be read via journalctl, logstash, or by looking at /var/log/magnum/magnum-api.log. There is seldom anything of interest in the logs but they may contain useful messages if there are fundamental integration issues between the different magnum components.

Typically a user will interact with magnum-api using opentofu or the openstack cli. In the cli, magnum is referred to as 'cluster coe,', for example:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
 # openstack coe cluster list
&lt;/syntaxhighlight&gt;

== Magnum-cluster-api ==
The [https://github.com/vexxhost/magnum-cluster-api cluster-api driver] is not packaged for Debian or part of the osbpo as of May, 2026, so we need to build it ourselves. It uses Rust so you'll need a newish 'rustc' package installed on your build host. You will also need [https://github.com/upciti/wheel2deb wheel2deb]. wheel2dev builds into a single standalone binary, so you may need to build it but you don't need to install it anywhere in particular.

Clone the repository, and build with ''maturin build'' in the top-level dir. Then wherever the wheel appears (probably 'magnum-cluster-api/target/wheels/output', run wheel2deb with no arguments. That should get you a .deb package suitable for adding to reprepro.

== Magnum-conductor ==
Magnum-conductor schedules and implements the various steps in cluster creation. It is installed and managed by puppet and runs on cloudcontrol nodes.

Most of the real work of creating a kubernetes cluster is delegated to cluster api, so magnum-conductor is mostly limited to maintaining and reporting cluster status.

Magnum-api and magnum-conductor are configured in /etc/magnum/magnum.conf on a cloudcontrol.

== Octavia ==

Cluster-api creates an Octavia loadbalancer as a front end to the created k8s cluster. As of version 'epoxy,' it insists on creating a floating IP as the entry point to that loadbalancer. That means that a project must have floating IP quota to create a cluster.

== Neutron ==

Typically we create clusters on an existing VM network (e.g. VXLAN/IPv6-dualstack).

== Nova ==

The actual k8s worker nodes managed by magnum will appear as regular nova VMs within the project of the magnum user. They typically have names like &quot;&lt;cluster-name-worker-1-2&gt;-&lt;id&gt;-default-worker-&lt;id&gt;&quot;


= Cluster-api (capi) Components =
== capi service cluster ==

Cluster api runs on a kubernetes cluster. In order to avoid chicken/egg concerns, the cluster that hosts capi is NOT managed by magnum. Instead, it runs on an easily-recreated single node k3s cluster and as described by a helm chart. This cluster can be created and configured with the puppet class 'profile::openstack::capi'

To build a new capi worker:

1. Create a new debian VM in the 'magnum' project named 'capi-worker-xxx'. One core and 2GB of ram seems sufficient.

2. Apply the 'profile::openstack::capi' class to that VM (there should already be a prefix config that does this), and allow puppet to stabilize (it will take two or three puppet runs). Puppet will create a k3s cluster and then apply a helm chart hosted on our chartmuseum repo to set up capi.

3. Apply security group rules that allow cloudcontrol access to port 644 (typically via a pre-existing service group like 'cloudcontrol-to-kubernetes')

4. Confirm that the worker is set up properly:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
labtestandrew@capi-worker-1:~$ export KUBECONFIG=/etc/rancher/k3s/k3s.yaml 
labtestandrew@capi-worker-1:~$ sudo kubectl get pods --all-namespaces
NAMESPACE                           NAME                                                              READY   STATUS    RESTARTS      AGE
capi-addon-system                   cluster-api-addon-provider-756bfd798f-5gpnk                       1/1     Running   0             44h
capi-janitor-system                 cluster-api-janitor-openstack-55fb64777c-d9bgb                    1/1     Running   0             44h
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-5b959f764c-cl79q        1/1     Running   6 (41h ago)   44h
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-7556869f8-ctl5d     1/1     Running   3 (43h ago)   44h
capi-system                         capi-controller-manager-5b67d4fc7-c94md                           1/1     Running   6 (41h ago)   44h
capo-system                         capo-controller-manager-6f899b5b7b-spslb                          1/1     Running   3 (41h ago)   44h
cert-manager                        cert-manager-7d67448f59-t5vlb                                     1/1     Running   0             44h
cert-manager                        cert-manager-cainjector-666b8b6b66-mmt5k                          1/1     Running   0             44h
cert-manager                        cert-manager-webhook-78cb4cf989-6xwfm                             1/1     Running   2 (43h ago)   44h
kube-system                         coredns-5688667fd4-9hsqt                                          1/1     Running   2 (43h ago)   45h
kube-system                         local-path-provisioner-774c6665dc-g5fld                           1/1     Running   0             45h
kube-system                         metrics-server-6f4c6675d5-8fw6r                                   1/1     Running   2 (43h ago)   45h
magnum-admin                        test-cluster-capi-worker-1-2-edez5vbbrppf-autoscaler-6bff5mbbnr   1/1     Running   5 (42m ago)   47m
&lt;/syntaxhighlight&gt;

5. Once that worker is up and running, you need to direct magnum to use the new cluster. As created by k3s, /etc/rancher/k3s/k3s.yaml  contains everything that Magnum will need to know with the exception of the host IP. Make a copy of k3s.yaml but replace 127.0.0.1 in the the 'server' line with the actual IP of the worker.

6. Add this existing k3s.yaml contents to private puppet, as modules/secret/secrets/openstack/&lt;codfw1dev/eqiad1&gt;/magnum/capiservicek3s.yaml -- it will then be installed on cloudcontrols.

== node images ==

Each magnum template will refer to a base VM image. These images should be special ready-made ubuntu or debian images, prebuilt with kubernetes already installer or partially configured. As of today, the best images to be used are built and published by vexxhost, found at https://github.com/vexxhost/capo-image-elements/releases.

= Troubleshooting =

Any logs about failure or success will be present only on the capi worker; the most useful ones are most likely written by the capo-controller-manager. To view them, log onto the capi worker VM and note capo-controller-manager name from

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
labtestandrew@capi-worker-1:~$ sudo kubectl get pods --all-namespaces
...
capi-system                         capi-controller-manager-5b67d4fc7-c94md                           1/1     Running   6 (41h ago)   44h
capo-system                         capo-controller-manager-6f899b5b7b-spslb                          1/1     Running   3 (41h ago)   44h
...
&lt;/syntaxhighlight&gt;

Then, to see the logs:
&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@capi-worker-1:~# kubectl  logs -n capo-system   capo-controller-manager-6f899b5b7b-spslb  | less
&lt;/syntaxhighlight&gt;

the capi-controller-manager logs may also be helpful.

= Other necessary services =
== image repo ==

We try not to build our control plain out of random upstream packages; rather, we mirror particular upstream versions in an internal repo.

The images used for creating the capi worker cluster (as well as the k3s images themselves) are stored on a VM in the cloudinfra project named 'docker-registry-XX' with the images themselves stored on a mounted cinder volume. These packages can be upgraded from various upstream repos; when version string changes the associated version numbers must also be updated in hiera with settings like 'profile::openstack::capi::cluster_api_version'

== helm chart repo ==

Similarly, we try to host our own helm charts for management of the cluster-api service cluster as well as for user-created magnum clusters.

Those charts are hosted on a simple chartmuseum instance in 'cloudinfra' named chartmuseum-X.

All charts are exact copies of upstream charts, with one important exception: we need to override the network ranges in the upstream magnum cluster chart in order to avoid conflict with out existing setup. That chart is (currently) called openstack-cluster.

Upstream charts can be checked out from the [https://github.com/azimuth-cloud/capi-helm-charts azimuth github repo]. There's also a working checkout of that repo in the /srv volume of the chartmuseum VM in cloudinfra.

If starting with a raw upstream checkout, first apply this patch to avoid network conflicts:

&lt;syntaxhighlight lang=&quot;diff&quot;&gt;
diff --git a/charts/openstack-cluster/values.yaml b/charts/openstack-cluster/values.yaml
index 6bad53b..7890d91 100644
--- a/charts/openstack-cluster/values.yaml
+++ b/charts/openstack-cluster/values.yaml
@@ -62,12 +62,16 @@ kubeNetwork:
   # By default, use the private network range 172.16.0.0/12 for the cluster network
   # We split it into two equally-sized blocks for pods and services
   # This gives ~500,000 addresses in each block
+  #
+  # WMF HACK: Upstream uses 172.16.0.0/12 but we need to use 10.16.0.0/12 to avoid
+  #  clashing with other bits of our Neutron setup
+  #
   pods:
     cidrBlocks:
-      - 172.16.0.0/13
+      - 10.16.0.0/13
   services:
     cidrBlocks:
-      - 172.24.0.0/13
+      - 10.24.0.0/13
   serviceDomain: cluster.local
 
 # Settings for the OpenStack networking for the cluster
@@ -187,7 +191,7 @@ apiServer:
   #  - 10.10.0.0/16  # IPv4 Internal Network
   #  - 123.123.123.123 # some other IPs
   # Indicates whether to associate a floating IP with the API server
-  associateFloatingIP: true
+  associateFloatingIP: false
   # The specific floating IP to associate with the API server
   # If not given, a new IP will be allocated if required
   floatingIP:
&lt;/syntaxhighlight&gt;


After fetching, rebasing, and &lt;optionally&gt; modifying, you can package and release the new chart like this:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
root@chartmuseum-2:/srv/capi-helm-charts/charts/openstack-cluster# helm dependency build
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the &quot;chartmuseum&quot; chart repository
...Successfully got an update from the &quot;chartmuseum.wmcloud.org&quot; chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Deleting outdated charts

root@chartmuseum-2:/srv/capi-helm-charts/charts/
openstack-cluster# cd ..
root@chartmuseum-2:/srv/capi-helm-charts/charts# helm package ./openstack-cluster --version 0.16.1-wmf
Successfully packaged chart and saved it to: /srv/capi-helm-charts/charts/openstack-cluster-0.16.1-wmf.tgz

root@chartmuseum-2:/srv/capi-helm-charts/charts# cp openstack-cluster-0.16.1-wmf.tgz /srv/chartmuseum/
&lt;/syntaxhighlight&gt;


To check the latest available chart, you can pull it back down from chartmuseum in a separate directory:

&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
$ mkdir checkthecharts
$ cd checkthecharts
$ helm repo add chartmuseum  https://chartmuseum.wmcloud.org
$ helm pull chartmuseum/openstack-cluster
&lt;/syntaxhighlight&gt;

== helm ==</text>
      <sha1>1iwfro39os01qdx2wfytxd3qujouvbx</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-monitoring/SAL</title>
    <ns>498</ns>
    <id>459243</id>
    <revision>
      <id>2414333</id>
      <parentid>2414093</parentid>
      <timestamp>2026-05-17T00:27:22Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)</comment>
      <origin>2414333</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="16265" sha1="a9ib6ng1phth3vzn06wz5y9s29geop6" xml:space="preserve">=== 2026-05-17 ===
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>a9ib6ng1phth3vzn06wz5y9s29geop6</sha1>
    </revision>
    <revision>
      <id>2414334</id>
      <parentid>2414333</parentid>
      <timestamp>2026-05-17T00:54:54Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)</comment>
      <origin>2414334</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="16474" sha1="ie9mvmkcqotkmt3hmcm3sws7668jdwe" xml:space="preserve">=== 2026-05-17 ===
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>ie9mvmkcqotkmt3hmcm3sws7668jdwe</sha1>
    </revision>
    <revision>
      <id>2414335</id>
      <parentid>2414334</parentid>
      <timestamp>2026-05-17T01:07:29Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)</comment>
      <origin>2414335</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="16683" sha1="gbdd6byrk000i1jobij00tv97gnnew1" xml:space="preserve">=== 2026-05-17 ===
* 01:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>gbdd6byrk000i1jobij00tv97gnnew1</sha1>
    </revision>
    <revision>
      <id>2414336</id>
      <parentid>2414335</parentid>
      <timestamp>2026-05-17T01:10:40Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977577391 (https://github.com/cluebotng/component-configs/commits/b7ea6acca0022ec902075282f7fd686162948bc8)</comment>
      <origin>2414336</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="16892" sha1="2xg4pgnobgnp81suv1vkthmh3roz15s" xml:space="preserve">=== 2026-05-17 ===
* 01:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977577391 (https://github.com/cluebotng/component-configs/commits/b7ea6acca0022ec902075282f7fd686162948bc8)
* 01:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>2xg4pgnobgnp81suv1vkthmh3roz15s</sha1>
    </revision>
    <revision>
      <id>2414340</id>
      <parentid>2414336</parentid>
      <timestamp>2026-05-17T08:50:36Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986256508 (https://github.com/cluebotng/component-configs/commits/fcc6e35d16a8ccd601ed7f37bbafcc9b26b8f80f)</comment>
      <origin>2414340</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="17101" sha1="6obwvok5siism3uivg4012rib96980d" xml:space="preserve">=== 2026-05-17 ===
* 08:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25986256508 (https://github.com/cluebotng/component-configs/commits/fcc6e35d16a8ccd601ed7f37bbafcc9b26b8f80f)
* 01:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977577391 (https://github.com/cluebotng/component-configs/commits/b7ea6acca0022ec902075282f7fd686162948bc8)
* 01:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977567974 (https://github.com/cluebotng/component-configs/commits/7f853bffd99230f195cb0bb08b2a29efcd929b17)
* 00:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25977310961 (https://github.com/cluebotng/component-configs/commits/febede844a3f1ef2246a9529152904e5d3f07272)
* 00:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25976823874 (https://github.com/cluebotng/component-configs/commits/e37f79c584a138b7783dc854dbf40e6338905e83)

=== 2026-05-14 ===
* 21:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18)
* 21:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0)
* 21:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f)
* 21:01 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b)

=== 2026-05-13 ===
* 01:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a)
* 01:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e)
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979)

=== 2026-05-12 ===
* 12:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25735536230 (https://github.com/cluebotng/component-configs/commits/245e38970666704881e64dbcdbf137cd7cf4769b)
* 12:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25733759682 (https://github.com/cluebotng/component-configs/commits/d71f9d5a70137a2c686d661ffd2e3ea4ce69115b)
* 11:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25731464158 (https://github.com/cluebotng/component-configs/commits/ae6d4cad7abfa0dd2bde0d04554e35bad6069ed1)
* 09:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25726905831 (https://github.com/cluebotng/component-configs/commits/cf3a002fc1808019d0018f4f38c1076cdbcaa643)
* 09:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724491283 (https://github.com/cluebotng/component-configs/commits/8291e582c787ae658abde8c26e8bc2ae9dd381fb)
* 08:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25722995016 (https://github.com/cluebotng/component-configs/commits/b4212e1e0296be2fd84c620a56c3d744d9cbaf30)

=== 2026-05-09 ===
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25607652745 (https://github.com/cluebotng/component-configs/commits/25ea91ed216c1cca5ca3de2da9abd7ac1337eb86)

=== 2026-05-06 ===
* 21:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462948750 (https://github.com/cluebotng/component-configs/commits/78047ed0bf8def9f4ff2b181d1710a274328a5b3)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25462408819 (https://github.com/cluebotng/component-configs/commits/c905103b2bd6129e76da6cf191567f82913c8349)
* 20:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460509016 (https://github.com/cluebotng/component-configs/commits/b9c2dde9382e1a3a5502f5218ecc7eaef6b4f9ec)
* 20:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25460310332 (https://github.com/cluebotng/component-configs/commits/511feb967af13f77e59803c69bc3447124733524)
* 18:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452753895 (https://github.com/cluebotng/component-configs/commits/6680b6abb8b5b775f79893981bd070c9e17f0358)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25452220343 (https://github.com/cluebotng/component-configs/commits/2ba80dd76efd2798ab7d1fedddf2a7baeab1118b)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451592437 (https://github.com/cluebotng/component-configs/commits/88790d34de5a0bca24d8685f3f7df749b052647b)
* 16:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448624822 (https://github.com/cluebotng/component-configs/commits/92c52e117c73389fe04c66d53ad083eef60ef387)
* 16:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25448365994 (https://github.com/cluebotng/component-configs/commits/7eaf56e56e3eacfd97455f271b152f369ca2b3f4)

=== 2026-04-25 ===
* 02:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919967572 (https://github.com/cluebotng/component-configs/commits/3940629dd68ec9bd151f28eece25db46955ccec8)
* 01:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24919765830 (https://github.com/cluebotng/component-configs/commits/a524bd86e0470289dfab9c755b32ffa9a8f71a42)
* 01:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24918981203 (https://github.com/cluebotng/component-configs/commits/4aec1980bbdc5a4fd34c73a044f672bb26a5af80)

=== 2026-04-24 ===
* 22:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24914156754 (https://github.com/cluebotng/component-configs/commits/693acb1af44127a03eaa151ac35b7894c69ab163)
* 20:48 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24911063126 (https://github.com/cluebotng/component-configs/commits/308ad964ff269a70a7aabfc1cb36e924360806a0)

=== 2026-04-17 ===
* 02:51 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24545107848 (https://github.com/cluebotng/component-configs/commits/93640977d7dd4117faf49bd179aac37720eee5b2)
* 01:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24543036687 (https://github.com/cluebotng/component-configs/commits/b97517784b752f7d0055837db122456017b21866)

=== 2026-04-16 ===
* 23:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24539805608 (https://github.com/cluebotng/component-configs/commits/bef895ecceca2ac8c0bafe1e39e4680142771a39)
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24536002079 (https://github.com/cluebotng/component-configs/commits/46f94a272d26bcd13cf55d2a223ba74fde716d4e)
* 21:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24535245433 (https://github.com/cluebotng/component-configs/commits/6b66d29ba7f99387d0fed0292e21c235b3f9ab0b)
* 12:42 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24510746155 (https://github.com/cluebotng/component-configs/commits/e927788672bbd71dc8a52dc4b34db54abe3995e9)
* 11:09 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24506838480 (https://github.com/cluebotng/component-configs/commits/db30680b466b66aed23b2f87989cbe2e9fd90511)
* 09:38 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502989226 (https://github.com/cluebotng/component-configs/commits/d1caa7e619b2f6df59cbdb32d5bdf150ceacfa19)

=== 2026-04-14 ===
* 21:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24424850045 (https://github.com/cluebotng/component-configs/commits/9124b4b266ce71985cca6d82fd4261c1850b6d4d)
* 21:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24422702845 (https://github.com/cluebotng/component-configs/commits/6d3be4a356e5c50103fa3381dc7c9ce81bf1788f)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442419 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897960 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 14:25 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24247609205 (https://github.com/cluebotng/component-configs/commits/51257ea555ac174e0fe397b1923edbf76fada3dc)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247598064 (https://github.com/cluebotng/component-configs/commits/7ec9cb1bdb9d3218d0388086118e3609ccb68956)

=== 2026-04-09 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206712245 (https://github.com/cluebotng/component-configs/commits/e7d5ec988541b9d441a5c565f624b7e88e11204f)
* 18:17 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24206093215 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659587521 (https://github.com/cluebotng/component-configs/commits/6d62fa6482ab7ce2ff8a99029633d738f068af76)
* 17:42 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659545398 (https://github.com/cluebotng/component-configs/commits/8bb9fa3878bfe2763bc9fc12643727602969ae50)

=== 2026-03-21 ===
* 16:22 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383640353 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383563872 (https://github.com/cluebotng/component-configs/commits/6bbc2343b2b24ab7149ad6471504f80a9d529d11)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383560207 (https://github.com/cluebotng/component-configs/commits/86937ba3c912f82970faef09c876b888834d96b2)

=== 2025-12-25 ===
* 13:43 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/20505969753 (https://github.com/cluebotng/component-configs/commits/5a347e229af9bc830ef9a03e0446b46dd8ccfb15)

=== 2025-12-01 ===
* 08:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)
* 08:43 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19816470332 (https://github.com/cluebotng/component-configs/commits/00278e339c41812ca8ecd179e1630abfb031117b)

=== 2025-11-12 ===
* 07:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18)

=== 2025-11-11 ===
* 15:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:36 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19270642940 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270294471 (https://github.com/cluebotng/component-configs/commits/df4e433c6a567df4484b7115ddf2c53fe1f9494f)
* 15:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270285003 (https://github.com/cluebotng/component-configs/commits/e103f6ac56b26a2d6e3c0705c81c75a4419287cc)
* 14:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19268985739 (https://github.com/cluebotng/component-configs/commits/f88bf173399c2591eca2357bbc9bff54ee70b731)

=== 2025-11-09 ===
* 19:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213633185 (https://github.com/cluebotng/component-configs/commits/f94671068b275a0195e1001943ca83f40b8d81f2)
* 19:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19213259803 (https://github.com/cluebotng/component-configs/commits/65af33a993b42c6821f5053c548a142d5b11a42d)

=== 2025-11-04 ===
* 11:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19066512964 (https://github.com/cluebotng/component-configs/commits/38bbff0654c639bfe723e48347ec173abc2a3c96)

=== 2025-11-02 ===
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016297587 (https://github.com/cluebotng/component-configs/commits/7620c10586a5803b6d797aac0fcee5303142d049)
* 18:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19016164140 (https://github.com/cluebotng/component-configs/commits/042c1033922105af1510ad83356d090383abb809)
* 17:54 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015945300 (https://github.com/cluebotng/component-configs/commits/b09ca1fd5dc19977fb34f5f85107e861e53587e7)
* 17:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015905691 (https://github.com/cluebotng/component-configs/commits/3cb78b899eadbb2f6060ea355e89b4c15be3919e)
* 17:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015890628 (https://github.com/cluebotng/component-configs/commits/9a7a2fa73af31eda4276cabe550fb40bf7c173a5)
* 17:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19015435727 (https://github.com/cluebotng/component-configs/commits/e834d58735fb8e7d28a81f91d23508869c73d1c5)

=== 2025-09-24 ===
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198507 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820840 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 17:06 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984009139 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983743046 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 18:08 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17924276736


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>6obwvok5siism3uivg4012rib96980d</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.cluebotng-trainer/SAL</title>
    <ns>498</ns>
    <id>459244</id>
    <revision>
      <id>2414339</id>
      <parentid>2413987</parentid>
      <timestamp>2026-05-17T08:32:02Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25985888959 (https://github.com/cluebotng/component-configs/commits/591f7870cefcc3c295b1136fa6afb385738dd863)</comment>
      <origin>2414339</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="17347" sha1="dgmchylfn3iskgxf64cabguikkl2p3z" xml:space="preserve">=== 2026-05-17 ===
* 08:32 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25985888959 (https://github.com/cluebotng/component-configs/commits/591f7870cefcc3c295b1136fa6afb385738dd863)

=== 2026-05-14 ===
* 17:02 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda)

=== 2026-05-13 ===
* 01:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771881181 (https://github.com/cluebotng/component-configs/commits/d180b98d98a8b5c89115a2080375145455ab4730)

=== 2026-05-12 ===
* 10:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25729275449 (https://github.com/cluebotng/component-configs/commits/ee80e1a674aa4d62ec0736286534130973827ac8)
* 09:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724848224 (https://github.com/cluebotng/component-configs/commits/8f3d84d5a91af96267b04964a47292b30c680076)
* 09:13 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25724842081 (https://github.com/cluebotng/component-configs/commits/8bc931f8c1f1c93df322457a7abadec867f9f46c)
* 08:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25723480610 (https://github.com/cluebotng/component-configs/commits/25c0a1035daa67c2225c0f7f7a414ff5cfb6ed2a)

=== 2026-05-06 ===
* 20:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25459711948 (https://github.com/cluebotng/component-configs/commits/3e0d938cc4bbeb5069a3e0b0d864df7eabf78c23)
* 17:56 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25451879961 (https://github.com/cluebotng/component-configs/commits/966f5e00a0a1098153ca8b6af11bc1124a119a1b)
* 16:35 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25447977391 (https://github.com/cluebotng/component-configs/commits/bd39eba234686d41e4d889def1ed91fc28e8fdf6)

=== 2026-04-24 ===
* 21:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24913659688 (https://github.com/cluebotng/component-configs/commits/c5ad64125c64f5db76d9a8141163d380f1c2d4f0)

=== 2026-04-20 ===
* 22:04 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24692640197 (https://github.com/cluebotng/component-configs/commits/6ccc54ff83a9536189370f93b0f3b7f9225e4259)

=== 2026-04-17 ===
* 14:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24569591292 (https://github.com/cluebotng/component-configs/commits/f80158cf5eb8b262e699d1bd27b4793819987bf6)
* 00:20 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24540900579 (https://github.com/cluebotng/component-configs/commits/bfd1f6b6b742e082dbbc587464ca135bfe827765)

=== 2026-04-16 ===
* 18:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24526007472 (https://github.com/cluebotng/component-configs/commits/37d3288f49b6baec42eb13403fb004de8b7e6271)
* 09:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24502903296 (https://github.com/cluebotng/component-configs/commits/09046ab7e1f4831d3acaf275abbb61a756291a24)

=== 2026-04-13 ===
* 17:33 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24357383991 (https://github.com/cluebotng/component-configs/commits/80829bf9249aa902be5209b117674eccf8283939)

=== 2026-04-10 ===
* 15:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24250442397 (https://github.com/cluebotng/component-configs/commits/bfa8b761a017e9b8bb69ae52c5cb731d17bd324f)
* 15:14 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249897956 (https://github.com/cluebotng/component-configs/commits/68514222ba9a90ece524baf75b02c9835faf87d3)
* 15:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24249371602 (https://github.com/cluebotng/component-configs/commits/acce8e3abd2da42aef50053d1c1126daec711da1)
* 14:49 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24248696057 (https://github.com/cluebotng/component-configs/commits/310efa6d212253bd549cddd16d85f4167b26f684)
* 14:27 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247621716 (https://github.com/cluebotng/component-configs/commits/2e25667a8eb92011273ca47b217f3546da48cae9)
* 14:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24247582931 (https://github.com/cluebotng/component-configs/commits/4c3786d55025827d330be1505ae5c35b0beed2b5)

=== 2026-04-09 ===
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24206093268 (https://github.com/cluebotng/component-configs/commits/a97bfe791582e24f1c696f1bd89b965ea233c253)

=== 2026-03-27 ===
* 17:44 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23659602695 (https://github.com/cluebotng/component-configs/commits/6f1ef446ee988193ccd8d9c62eb4841267976159)
* 17:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23659537319 (https://github.com/cluebotng/component-configs/commits/c3f980e28e95bd1081b2ed9c903d2ac4d51b2c3b)

=== 2026-03-21 ===
* 16:39 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/23383959549 (https://github.com/cluebotng/component-configs/commits/93c7795074704e8f5e28a9664a00d04cfd26c09c)
* 16:26 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:23 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383640343 (https://github.com/cluebotng/component-configs/commits/3497a25c3d209bdf8f64f3ec3e77e52f2f8debfa)
* 16:18 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383556336 (https://github.com/cluebotng/component-configs/commits/88e098b5a6669f94800824e05af8f39882274193)
* 16:13 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23383552275 (https://github.com/cluebotng/component-configs/commits/3922e765a155e2a966a94aff013d2f0ad4c0dd43)

=== 2025-11-14 ===
* 08:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358864102 (https://github.com/cluebotng/component-configs/commits/b9059392c8da641c97c97919e1d017bf0251914f)
* 08:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19358733947 (https://github.com/cluebotng/component-configs/commits/80f11fd31e068d517962c3bcc76a29581d4d3e0a)

=== 2025-11-13 ===
* 22:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19347680045 (https://github.com/cluebotng/component-configs/commits/5de2267efa622608c868e4cb66658a4f367aa3b0)
* 21:49 wm-bot2: Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/19346862269 (https://github.com/cluebotng/component-configs/commits/65fa127d349eb5521d47017350bdc52ceb35d9ae)
* 19:15 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342998586 (https://github.com/cluebotng/component-configs/commits/17a2abf82e9a2076113891e4ae3d5159c048b940)
* 19:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342782944 (https://github.com/cluebotng/component-configs/commits/fed3ef035303fd54334c5a7932d23a43e381492e)
* 18:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19342079797 (https://github.com/cluebotng/component-configs/commits/b8c173ced436297aaa9a09b2ef9a4b814d696a30)
* 18:29 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341777629 (https://github.com/cluebotng/component-configs/commits/afb04303301910afc2a7692e2e5676540a27c38c)
* 18:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341671161 (https://github.com/cluebotng/component-configs/commits/84f054bb9402daea356833fbd2d606f4d2b44b48)
* 18:17 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341445508 (https://github.com/cluebotng/component-configs/commits/2a497ee04f7182e6d114d1aafa79a99e29ff6d22)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341306596 (https://github.com/cluebotng/component-configs/commits/6e3dd964163e25f7115b74c7b132316745613729)
* 18:03 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19341017921 (https://github.com/cluebotng/component-configs/commits/b28d25e5b98fe60c2b85138e7d1a2a52cee03383)
* 17:59 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340976225 (https://github.com/cluebotng/component-configs/commits/8c3a4f743978487c10274534162bb1a7a2ceb8a8)
* 17:50 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340717614 (https://github.com/cluebotng/component-configs/commits/a2ac5e2b12d80df59f7f37209ed8943fe471992a)
* 17:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19340563565 (https://github.com/cluebotng/component-configs/commits/a5ea7ec11c7d2a7ced4f29df1f2a742f52dbf777)
* 15:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19337204923 (https://github.com/cluebotng/component-configs/commits/d5247b0e2c72c4426888761d70d73f59ba7362a5)

=== 2025-11-11 ===
* 15:37 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270642882 (https://github.com/cluebotng/component-configs/commits/3fe913812986e82db75d4a6657cba3f697f5649c)
* 15:28 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270313845 (https://github.com/cluebotng/component-configs/commits/900a30b8983bc80a537330fa4345e7952f2081b4)
* 15:25 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19270264807 (https://github.com/cluebotng/component-configs/commits/531100511571d8262fc874f0f4cddfa2df3d6c2b)
* 13:44 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19267535662 (https://github.com/cluebotng/component-configs/commits/eae37aa10505a2ac3c773cdd4f85b7222cdae836)
* 13:24 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19266909480 (https://github.com/cluebotng/component-configs/commits/bc0dd19078113c4e1cbe90a059fe26932fb70a29)

=== 2025-10-07 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18304356335 (https://github.com/cluebotng/component-configs/commits/49f902fb1380f69c5e0156466832b031a9c228ee)

=== 2025-10-06 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18272376354 (https://github.com/cluebotng/component-configs/commits/449c7d3c68ade43ce238b9b335182dc50f929511)

=== 2025-10-05 ===
* 06:41 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18255175939 (https://github.com/cluebotng/component-configs/commits/f7e65671f2deb536f52fdf0c1db7ec80709a76da)

=== 2025-10-01 ===
* 06:46 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18153820283 (https://github.com/cluebotng/component-configs/commits/9f9e9abc422c8a74207416afbaf81b72509e91ce)

=== 2025-09-30 ===
* 06:47 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18121221077 (https://github.com/cluebotng/component-configs/commits/11eb3da0c88160bca0ec60e55dadad7e928e40f3)

=== 2025-09-29 ===
* 16:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101421 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5)
* 15:21 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101913411 (https://github.com/cluebotng/component-configs/commits/1b6389ae74b8974d6f49591c0abf14a8da974c4b)
* 15:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460176 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4)

=== 2025-09-27 ===
* 13:10 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18060217564 (https://github.com/cluebotng/component-configs/commits/049d68d8af679276a08e05d41c90210efa03240b)

=== 2025-09-26 ===
* 06:45 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030092297 (https://github.com/cluebotng/component-configs/commits/2516589b458034b7c2540c998b29debeb63e82df)

=== 2025-09-25 ===
* 16:58 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801857 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c)
* 16:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221974 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d)
* 16:16 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782304 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e)
* 16:12 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677303 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a)
* 16:07 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531079 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f)
* 14:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008148799 (https://github.com/cluebotng/component-configs/commits/0bb2ca770839346853b6a8440927a8afde70c94d)

=== 2025-09-24 ===
* 18:36 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986145339 (https://github.com/cluebotng/component-configs/commits/820291b11edc6256cfa07e9a4a73677df86e52d8)
* 18:34 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17986015781 (https://github.com/cluebotng/component-configs/commits/b0f088ec49a2156a18ad7d78d18640f6e8fe943c)
* 18:31 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985989347 (https://github.com/cluebotng/component-configs/commits/38b2f2235d143f0522d560a93c04e50e8fa38c77)
* 18:19 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985728221 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:11 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985446508 (https://github.com/cluebotng/component-configs/commits/34320f16093c0d4160ce346108bc14bfb95245f8)
* 18:08 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985386789 (https://github.com/cluebotng/component-configs/commits/a1633fc7ecf1bd4a310fcb03d02d0da81212427a)
* 18:05 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985333827 (https://github.com/cluebotng/component-configs/commits/24f56031bca40f808e05920eb128fd892a3d9b92)
* 17:57 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985198515 (https://github.com/cluebotng/component-configs/commits/cfa2541734b05a9da326bbeab2e82cc21d6e91e4)
* 17:52 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17985077117 (https://github.com/cluebotng/component-configs/commits/955cd61533af834d56d36117a81643d8ab9ba81f)
* 17:40 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17984820831 (https://github.com/cluebotng/component-configs/commits/6f47ae931d95d85e2c3c1d6b42f1eabc6d3b1960)
* 16:55 wm-bot2: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17983737370 (https://github.com/cluebotng/component-configs/commits/refs/heads/main)

=== 2025-09-22 ===
* 20:30 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17927644810
* 19:02 wmbot~damian-scripts@tools-bastion-15: Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/17925536572


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>dgmchylfn3iskgxf64cabguikkl2p3z</sha1>
    </revision>
  </page>
  <page>
    <title>Category:SCROLL reviews</title>
    <ns>14</ns>
    <id>460181</id>
    <revision>
      <id>2414312</id>
      <timestamp>2026-05-16T19:49:23Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <comment>Created page with &quot;[[Category:SCROLLs|Reviews]]&quot;</comment>
      <origin>2414312</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="28" sha1="t89x6lzmtp74yx54c1o6rp09v7ghunz" xml:space="preserve">[[Category:SCROLLs|Reviews]]</text>
      <sha1>t89x6lzmtp74yx54c1o6rp09v7ghunz</sha1>
    </revision>
  </page>
  <page>
    <title>User:Koavf/common.js</title>
    <ns>2</ns>
    <id>460182</id>
    <revision>
      <id>2414314</id>
      <timestamp>2026-05-16T19:53:39Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <comment>from en.wikt</comment>
      <origin>2414314</origin>
      <model>javascript</model>
      <format>text/javascript</format>
      <text bytes="8471" sha1="cnjdtvjr70rh9dgl2dbij7logpdptqa" xml:space="preserve">mw.loader.using(['mediawiki.api', 'mediawiki.Title'], function () {
    &quot;use strict&quot;;

    var config = mw.config.get(['wgNamespaceNumber', 'wgTitle', 'wgUserGroups', 'skin']);

    function removeBlanks(arr) {
        var ret = [];
        var i, len;
        for (i = 0, len = arr.length; i &lt; len; i++) {
            var s = arr[i];
            s = s.trim();
            if (s) {
                ret.push(s);
            }
        }
        return ret;
    }

    function doMassDelete() {
        document.getElementById(&quot;wpMassDeleteSubmit&quot;).disabled = true;
        var articles = document.getElementById(&quot;wpMassDeletePages&quot;).value.split(&quot;\n&quot;);
        articles = removeBlanks(articles);
        if (!articles.length) {
            return;
        }
        var
            api = new mw.Api(),
            wpMassDeleteReasons = document.getElementById(&quot;wpMassDeleteReasons&quot;).value,
            wpMassDeleteReason = document.getElementById(&quot;wpMassDeleteReason&quot;).value,
            deleted = 0,
            failed = [],
            error = [],
            deferreds = [],
            reason = wpMassDeleteReasons == &quot;other&quot; ?
                wpMassDeleteReason :
                wpMassDeleteReasons + (wpMassDeleteReason ? &quot; (&quot; + wpMassDeleteReason + &quot;)&quot; : &quot;&quot;),
            onSuccess = function () {
                deleted++;
                document.getElementById(&quot;wpMassDeleteSubmit&quot;).value = &quot;(&quot; + deleted + &quot;)&quot;;
            };

        function makeDeleteFunc(article) {
            return function () {
                return $.Deferred(function (deferred) {
                    var promise = api.postWithToken('delete', {
                        format: 'json',
                        action: 'delete',
                        title: article,
                        reason: reason
                    });
                    promise.done(onSuccess);
                    promise.fail(function (code, obj) {
                        failed.push(article);
                        error.push(obj.error.info);
                    });
                    promise.always(function () {
                        deferred.resolve();
                    });
                });
            };
        }

        // Make a chain of deferred objects. We chain them rather than execute them in
        // parallel so that we don't make 1000 simultaneous delete requests and bring the
        // site down. We use deferred objects rather than the promise objects returned
        // from the API request so that the chain continues even if some articles gave
        // errors.
        var deferred = makeDeleteFunc(articles[0])();
        for (var i = 1, len = articles.length; i &lt; len; i++) {
            deferred = deferred.then(makeDeleteFunc(articles[i]));
        }

        // Show the output and do cleanup once all the requests are done.
        $.when(deferred).then(function () {
            document.getElementById(&quot;wpMassDeleteSubmit&quot;).value = &quot;Done (&quot; + deleted + &quot;)&quot;;
            if (failed.length) {
                var $failedList = $('&lt;ul&gt;');
                for(var x = 0; x &lt; failed.length; x++) {
                    // Link the titles in the &quot;failed&quot; array
                    var failedTitle = mw.Title.newFromText(failed[x]);
                    var $failedItem = $('&lt;li&gt;');
                    if (failedTitle) {
                        $failedItem.append( $('&lt;a&gt;')
                            .attr('href', failedTitle.getUrl())
                            .text(failed[x])
                        );
                    } else {
                        $failedItem.text(failed[x]);
                    }
                    $failedItem.append(document.createTextNode(': ' + error[x]));
                    $failedList.append($failedItem);
                }
                $('#wpMassDeleteFailedContainer')
                    .append($('&lt;br /&gt;'))
                    .append($('&lt;b&gt;')
                        .text('Failed deletions:')
                    )
                    .append($failedList);
            }
        });
    }
     
    function massdeleteform() {
        var bodyContent = (config.skin == &quot;cologneblue&quot; ? &quot;article&quot; : &quot;bodyContent&quot;);
        document.getElementsByTagName(&quot;h1&quot;)[0].textContent = &quot;Mass delete pages&quot;;
        document.title = &quot;Mass delete pages&quot;;
        document.getElementById(bodyContent).innerHTML = '&lt;h3 id=&quot;siteSub&quot;&gt;Lifted From Wikimedia Incubator - revision of 2015-10-28&lt;/h3&gt;&lt;br /&gt;' +
        '&lt;div&gt;' +
            '&lt;form id=&quot;wpMassDelete&quot; name=&quot;wpMassDelete&quot;&gt;' +
            '&lt;p&gt;Welcome to the mass deletion form, the page which you can use to mass delete a couple of pages. If you are deleting multiple pages that were created by one user, please use &lt;a href=&quot;/wiki/Special:Nuke&quot;&gt;Special:Nuke&lt;/a&gt;!&lt;/p&gt;' +
            '&lt;p&gt;If you are deleting &lt;b&gt;more than 10 pages&lt;/b&gt;, please &lt;a href=&quot;/wiki/Special:UserRights&quot;&gt;flag yourself&lt;/a&gt; as a bot first to avoid flooding &lt;a href=&quot;/wiki/Special:RecentChanges&quot;&gt;Special:RecentChanges&lt;/a&gt;, thank you!&lt;/p&gt;' +
            '&lt;p&gt;Please add the list of pages you would like to delete in the text box below, listing one page in one line. Thank you for using this tool!&lt;/p&gt;' +
            '&lt;div id=&quot;wpMassDeleteFailedContainer&quot;&gt;&lt;/div&gt;' +
            '&lt;br /&gt;' +
                'Pages to delete (one on each line, please):&lt;br /&gt;' +
                    '&lt;textarea tabindex=&quot;1&quot; accesskey=&quot;,&quot; name=&quot;wpMassDeletePages&quot; id=&quot;wpMassDeletePages&quot; rows=&quot;15&quot; cols=&quot;80&quot;&gt;&lt;/textarea&gt;' +
                '&lt;br /&gt;&lt;table style=&quot;background-color:transparent&quot;&gt;' +
                    '&lt;tr&gt;&lt;td&gt;Common reasons:&lt;/td&gt;' +
                        '&lt;td&gt;&lt;select id=&quot;wpMassDeleteReasons&quot;&gt;' +
                            '&lt;optgroup label=&quot;Other reason&quot;&gt;' +
                                '&lt;option value=&quot;other&quot;&gt;Other reason&lt;/option&gt;' +
                            '&lt;/optgroup&gt;' +
                            '&lt;optgroup label=&quot;Criteria for speedy deletion&quot;&gt;' +
                                '&lt;optgroup label=&quot;Common delete reasons&quot;&gt;' +
                                    '&lt;option value=&quot;Author request&quot;&gt;Author request&lt;/option&gt;' +
                                    '&lt;option value=&quot;Beyond scope&quot;&gt;Beyond scope&lt;/option&gt;' +
                                    '&lt;option value=&quot;Copyright violation&quot;&gt;Copyright violation&lt;/option&gt;' +
                                    '&lt;option value=&quot;Unneeded redirect&quot;&gt;Unneeded redirect(s)&lt;/option&gt;' +
                                    '&lt;option value=&quot;Vandalism&quot;&gt;Vandalism&lt;/option&gt;' +
                                '&lt;/optgroup&gt;' +
                                '&lt;optgroup label=&quot;Routine maintenance&quot;&gt;' +
                                    '&lt;option value=&quot;Process deletion&quot;&gt;Process deletion&lt;/option&gt;' +
                                    '&lt;option value=&quot;Orphaned talkpage&quot;&gt;Orphaned talkpage&lt;/option&gt;' +
                                    '&lt;option value=&quot;Redundant/extraneous after bulk-action&quot;&gt;Redundant/extraneous after bulk-action&lt;/option&gt;' +
                                    '&lt;option value=&quot;Source file status change&quot;&gt;Source file status change&lt;/option&gt;' +
                                    '&lt;option value=&quot;Superior transcription exists&quot;&gt;Superior transcription exists&lt;/option&gt;' +
                                '&lt;/optgroup&gt;' +
                            '&lt;/optgroup&gt;' +
                        '&lt;/select&gt;&lt;/td&gt;&lt;/tr&gt;' +
                    '&lt;tr&gt;&lt;td&gt;Other/additional reason:&lt;/td&gt;' +
                    '&lt;td&gt;&lt;input type=&quot;text&quot; id=&quot;wpMassDeleteReason&quot; name=&quot;wpMassDeleteReason&quot; size=&quot;42&quot; maxlength=&quot;255&quot; /&gt;&lt;/td&gt;&lt;/tr&gt;' +
                    '&lt;tr&gt;&lt;td&gt;&lt;input type=&quot;button&quot; id=&quot;wpMassDeleteSubmit&quot; name=&quot;wpMassDeleteSubmit&quot; value=&quot;Delete&quot; /&gt;&lt;/td&gt;' +
            '&lt;/form&gt;' +
        '&lt;/div&gt;';
        document.getElementById(&quot;wpMassDeleteReasons&quot;).onchange = function() {
            var maxlength = (document.getElementById(&quot;wpMassDeleteReasons&quot;).value == &quot;other&quot; ? 255 : 252-document.getElementById(&quot;wpMassDeleteReasons&quot;).value.length); //It's 252 because of the three characters (&quot; ()&quot;) in addition to the selected summary.
            document.getElementById(&quot;wpMassDeleteReason&quot;).setAttribute(&quot;maxlength&quot;, maxlength);
        };
        document.getElementById(&quot;wpMassDeleteSubmit&quot;).addEventListener(&quot;click&quot;, function (e) {
            doMassDelete();
        });
    }
     
    if (config.wgNamespaceNumber == -1 &amp;&amp;
        config.wgTitle.toLowerCase() == &quot;massdelete&quot; &amp;&amp;
        /sysop/.test(config.wgUserGroups)
    ) {
        massdeleteform();
    }

});</text>
      <sha1>cnjdtvjr70rh9dgl2dbij7logpdptqa</sha1>
    </revision>
  </page>
  <page>
    <title>MediaWiki:Jsonconfig-use-category</title>
    <ns>8</ns>
    <id>460183</id>
    <revision>
      <id>2414315</id>
      <timestamp>2026-05-16T19:56:33Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <comment>Removing unneeded cat</comment>
      <origin>2414315</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1" sha1="6za6z27cqk2qatlebqsiz85czbjf3rd" xml:space="preserve">-</text>
      <sha1>6za6z27cqk2qatlebqsiz85czbjf3rd</sha1>
    </revision>
  </page>
  <page>
    <title>Category:TrafficRunbooks</title>
    <ns>14</ns>
    <id>460184</id>
    <revision>
      <id>2414320</id>
      <timestamp>2026-05-16T20:00:21Z</timestamp>
      <contributor>
        <username>Koavf</username>
        <id>1382</id>
      </contributor>
      <comment>Created page with &quot;[[Category:Runbooks]]&quot;</comment>
      <origin>2414320</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="21" sha1="fqtc1h5ok8kgaqje534juvwryhuvyl4" xml:space="preserve">[[Category:Runbooks]]</text>
      <sha1>fqtc1h5ok8kgaqje534juvwryhuvyl4</sha1>
    </revision>
  </page>
</mediawiki>