Server Admin Log

<mediawiki xmlns="http://www.mediawiki.org/xml/export-0.11/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mediawiki.org/xml/export-0.11/ http://www.mediawiki.org/xml/export-0.11.xsd" version="0.11" xml:lang="en">
  <siteinfo>
    <sitename>Wikitech</sitename>
    <dbname>labswiki</dbname>
    <base>https://wikitech.wikimedia.org/wiki/Main_Page</base>
    <generator>MediaWiki 1.47.0-wmf.9</generator>
    <case>first-letter</case>
    <namespaces>
      <namespace key="-2" case="first-letter">Media</namespace>
      <namespace key="-1" case="first-letter">Special</namespace>
      <namespace key="0" case="first-letter" />
      <namespace key="1" case="first-letter">Talk</namespace>
      <namespace key="2" case="first-letter">User</namespace>
      <namespace key="3" case="first-letter">User talk</namespace>
      <namespace key="4" case="first-letter">Wikitech</namespace>
      <namespace key="5" case="first-letter">Wikitech talk</namespace>
      <namespace key="6" case="first-letter">File</namespace>
      <namespace key="7" case="first-letter">File talk</namespace>
      <namespace key="8" case="first-letter">MediaWiki</namespace>
      <namespace key="9" case="first-letter">MediaWiki talk</namespace>
      <namespace key="10" case="first-letter">Template</namespace>
      <namespace key="11" case="first-letter">Template talk</namespace>
      <namespace key="12" case="first-letter">Help</namespace>
      <namespace key="13" case="first-letter">Help talk</namespace>
      <namespace key="14" case="first-letter">Category</namespace>
      <namespace key="15" case="first-letter">Category talk</namespace>
      <namespace key="110" case="first-letter">Obsolete</namespace>
      <namespace key="111" case="first-letter">Obsolete talk</namespace>
      <namespace key="112" case="first-letter">OfficeIT</namespace>
      <namespace key="113" case="first-letter">OfficeIT talk</namespace>
      <namespace key="116" case="first-letter">Tool</namespace>
      <namespace key="117" case="first-letter">Tool talk</namespace>
      <namespace key="498" case="first-letter">Nova Resource</namespace>
      <namespace key="499" case="first-letter">Nova Resource Talk</namespace>
      <namespace key="666" case="first-letter">Heira</namespace>
      <namespace key="667" case="first-letter">Heira Talk</namespace>
      <namespace key="710" case="first-letter">TimedText</namespace>
      <namespace key="711" case="first-letter">TimedText talk</namespace>
      <namespace key="828" case="first-letter">Module</namespace>
      <namespace key="829" case="first-letter">Module talk</namespace>
    </namespaces>
  </siteinfo>
  <page>
    <title>Server Admin Log</title>
    <ns>0</ns>
    <id>7919</id>
    <revision>
      <id>2433179</id>
      <parentid>2433164</parentid>
      <timestamp>2026-07-05T02:00:22Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image</comment>
      <origin>2433179</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="108267" sha1="jf9gi35rmcqt76geo5bsqxeevnqwkqt" xml:space="preserve">
== 2026-07-05 ==
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-04 ==
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 58s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-03 ==
* 17:08 topranks: revert protocol preference changes on cr3-ulsfo after upgrade
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr2-eqord with reason: upgrade JunOS cr3-ulsfo
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr4-ulsfo with reason: upgrade JunOS cr3-ulsfo
* 16:48 topranks: reboot cr3-ulsfo to upgrade JunOS and reset linecard [[phab:T424839|T424839]]
* 15:52 topranks: adjust outbound BGP policies on cr3-ulsfo to drain router of traffic [[phab:T424839|T424839]]
* 15:45 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on lvs[4008-4010].ulsfo.wmnet with reason: upgrade JunOS cr3-ulsfo
* 15:44 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on asw1-[22-23]-ulsfo,cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt with reason: upgrade JunOS cr3-ulsfo
* 15:36 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 14:40 cmooney@dns3003: END - running authdns-update
* 14:26 cmooney@dns3003: START - running authdns-update
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:19 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:16 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 13:38 sukhe@dns1004: END - running authdns-update
* 13:35 sukhe@dns1004: START - running authdns-update
* 13:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:26 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:18 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:17 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:14 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:14 moritzm: imported samplicator 1.3.8rc1-1+deb13u1 to trixie-wikimedia/main [[phab:T337208|T337208]]
* 13:13 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:07 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:07 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:02 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:02 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:00 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:58 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:53 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:52 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:50 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 12:50 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 12:47 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:41 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:39 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:32 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 12:26 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 12:23 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 12:19 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[2004-2007].codfw.wmnet
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:15 jynus@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:09 jynus@cumin2003: START - Cookbook sre.dns.netbox
* 11:58 jynus@cumin2003: START - Cookbook sre.hosts.decommission for hosts backup[2004-2007].codfw.wmnet
* 10:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[1004-1007].eqiad.wmnet
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 10:01 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 09:52 jynus@cumin1003: START - Cookbook sre.dns.netbox
* 09:39 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:36 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts backup[1004-1007].eqiad.wmnet
* 09:36 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 09:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 09:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:57 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:55 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 08:49 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 08:49 atsukoito: depooling cirrussearch in codfw because of regression after upgrade [[phab:T431091|T431091]]
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mirror1001.wikimedia.org
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:29 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:18 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 08:11 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts mirror1001.wikimedia.org
* 06:15 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 18s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-02 ==
* 22:55 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1003.wikimedia.org with OS trixie
* 22:29 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:23 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:05 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint1003.wikimedia.org with OS trixie
* 22:03 mutante: contint1003 (zuul.wikimedia.org) - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 22:03 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on zuul.wikimedia.org with reason: reimage
* 21:39 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 18s)
* 21:39 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 21:20 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 21:19 sbassett: Deployed security fix for [[phab:T428829|T428829]]
* 20:58 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:55 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:40 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] (duration: 12m 35s)
* 20:36 arlolra@deploy1003: cscott, arlolra: Continuing with deployment
* 20:35 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 20s)
* 20:35 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 20:33 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint2003.wikimedia.org with OS trixie
* 20:31 arlolra@deploy1003: cscott, arlolra: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Cha
* 20:28 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]]
* 20:17 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] (duration: 08m 13s)
* 20:14 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:13 sbassett@deploy1003: sbassett: Continuing with deployment
* 20:11 sbassett@deploy1003: sbassett: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:09 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]]
* 20:08 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 20:08 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 20:08 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:05 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 19:49 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint2003.wikimedia.org with OS trixie
* 19:48 mutante: contint2003 - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 18:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:13 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 17:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1003.eqiad.wmnet with OS bookworm
* 17:52 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:52 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:51 jasmine@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=wikikube-ctrl1005.eqiad.wmnet
* 17:48 jasmine_: homer &quot;cr*eqiad*&quot; commit &quot;Added new stacked control plane wikikube-ctrl1005&quot;
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
* 17:31 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] (duration: 09m 33s)
* 17:26 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 17:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:23 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:21 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]]
* 17:18 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:16 rscout@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:15 rscout@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:12 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on wcqs[2002-2003].codfw.wmnet,wcqs1002.eqiad.wmnet with reason: reimaging hosts
* 17:08 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 17:07 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 17:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:03 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:03 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs1003
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs1003
* 17:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1003.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:57 tappof: bump space for prometheus k8s-aux in eqiad
* 16:55 cmooney@dns3003: END - running authdns-update
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:55 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:53 cmooney@dns3003: START - running authdns-update
* 16:52 ryankemper: [ml-serve-eqiad] Cleared out 1302 failed (Evicted) pods: `kubectl -n llm delete pods --field-selector=status.phase=Failed`, freeing calico-kube-controllers from OOM crashloop (evictions were caused by disk pressure)
* 16:49 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:46 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 16:39 rzl@dns1004: END - running authdns-update
* 16:37 rzl@dns1004: START - running authdns-update
* 16:36 rzl@dns1004: START - running authdns-update
* 16:35 rzl@deploy1003: Finished scap sync-world: [[phab:T416623|T416623]] (duration: 10m 19s)
* 16:34 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:33 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:30 rzl@deploy1003: rzl: Continuing with deployment
* 16:28 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:26 rzl@deploy1003: rzl: [[phab:T416623|T416623]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:25 rzl@deploy1003: Started scap sync-world: [[phab:T416623|T416623]]
* 16:25 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: sync
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: sync
* 16:16 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1003.eqiad.wmnet with OS bookworm
* 16:11 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:11 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Security updates
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 16:08 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 16:08 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Security updates
* 15:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Security updates
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:45 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:45 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Security updates
* 15:42 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1003.eqiad.wmnet with OS bookworm
* 15:24 moritzm: installing busybox updates from bookworm point release
* 15:20 moritzm: installing busybox updates from trixie point release
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Security updates
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:15 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:15 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Security updates
* 15:13 moritzm: installing giflib security updates
* 15:08 moritzm: installing Tomcat security updates
* 14:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 14:56 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 14:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:53 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Security updates
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 14:53 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 14:53 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Security updates
* 14:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:52 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94711 and previous config saved to /var/cache/conftool/dbconfig/20260702-144644-fceratto.json
* 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94709 and previous config saved to /var/cache/conftool/dbconfig/20260702-143636-fceratto.json
* 14:32 moritzm: installing libdbi-perl security updates
* 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94708 and previous config saved to /var/cache/conftool/dbconfig/20260702-142628-fceratto.json
* 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94707 and previous config saved to /var/cache/conftool/dbconfig/20260702-141621-fceratto.json
* 14:12 moritzm: installing rsync security updates
* 14:11 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94706 and previous config saved to /var/cache/conftool/dbconfig/20260702-140959-fceratto.json
* 14:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2205.codfw.wmnet with reason: Maintenance
* 14:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 14:07 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 Tran: Deployed patch for [[phab:T427287|T427287]]
* 14:04 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:59 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 13:59 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2205 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94704 and previous config saved to /var/cache/conftool/dbconfig/20260702-135505-fceratto.json
* 13:54 moritzm: installing sed security updates
* 13:53 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:52 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2209 to s3 primary [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94703 and previous config saved to /var/cache/conftool/dbconfig/20260702-135235-fceratto.json
* 13:52 federico3: Starting s3 codfw failover from db2205 to db2209 - [[phab:T430912|T430912]]
* 13:51 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:51 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2209 with weight 0 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94702 and previous config saved to /var/cache/conftool/dbconfig/20260702-134719-fceratto.json
* 13:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 [[phab:T430912|T430912]]
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:40 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:37 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 13:36 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:36 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:34 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 13:30 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:26 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:25 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
* 13:23 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:22 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 13:17 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:17 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns1004.wikimedia.org
* 13:12 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart (exit_code=97) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=97) rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:09 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] (duration: 07m 20s)
* 13:05 aude@deploy1003: jdrewniak, aude: Continuing with deployment
* 13:04 aude@deploy1003: jdrewniak, aude: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:02 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]]
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs-categories1001.eqiad.wmnet
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 12:10 jmm@dns1004: END - running authdns-update
* 12:07 jmm@dns1004: START - running authdns-update
* 11:51 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 11:44 btullis@cumin1003: START - Cookbook sre.dns.netbox
* 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:42 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 11:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 11:39 btullis@cumin1003: START - Cookbook sre.hosts.decommission for hosts wdqs-categories1001.eqiad.wmnet
* 11:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 11:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 11:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 11:29 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:29 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 10:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2214: Repooling
* 10:49 jmm@dns1004: END - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94698 and previous config saved to /var/cache/conftool/dbconfig/20260702-103146-fceratto.json
* 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94696 and previous config saved to /var/cache/conftool/dbconfig/20260702-102137-fceratto.json
* 10:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb1017.eqiad.wmnet
* 10:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 10:18 fceratto@cumin1003: Removing es1033 from zarcillo [[phab:T408772|T408772]]
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1033.eqiad.wmnet
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:14 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:13 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1017.eqiad.wmnet
* 10:12 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling
* 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94693 and previous config saved to /var/cache/conftool/dbconfig/20260702-101130-fceratto.json
* 10:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:10 fceratto@cumin1003: START - Cookbook sre.dns.netbox
* 10:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:03 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts es1033.eqiad.wmnet
* 10:03 fceratto@cumin1003: START - Cookbook sre.mysql.decommission
* 10:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94691 and previous config saved to /var/cache/conftool/dbconfig/20260702-100122-fceratto.json
* 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94690 and previous config saved to /var/cache/conftool/dbconfig/20260702-095529-fceratto.json
* 09:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 09:54 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:53 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2213: Repooling after switchover
* 09:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:44 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2213 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94688 and previous config saved to /var/cache/conftool/dbconfig/20260702-093859-fceratto.json
* 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94687 and previous config saved to /var/cache/conftool/dbconfig/20260702-093650-fceratto.json
* 09:36 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T430923|T430923]]
* 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94686 and previous config saved to /var/cache/conftool/dbconfig/20260702-093004-fceratto.json
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2192 with weight 0 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94685 and previous config saved to /var/cache/conftool/dbconfig/20260702-092455-fceratto.json
* 09:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s5 [[phab:T430923|T430923]]
* 09:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94684 and previous config saved to /var/cache/conftool/dbconfig/20260702-091957-fceratto.json
* 09:16 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] (duration: 06m 57s)
* 09:13 moritzm: installing libgcrypt20 security updates
* 09:12 kharlan@deploy1003: kharlan: Continuing with deployment
* 09:11 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94683 and previous config saved to /var/cache/conftool/dbconfig/20260702-090950-fceratto.json
* 09:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]]
* 09:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:01 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] (duration: 07m 07s)
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94682 and previous config saved to /var/cache/conftool/dbconfig/20260702-085942-fceratto.json
* 08:57 kharlan@deploy1003: kharlan: Continuing with deployment
* 08:56 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]]
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94681 and previous config saved to /var/cache/conftool/dbconfig/20260702-085237-fceratto.json
* 08:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 08:43 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:40 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:25 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] (duration: 11m 44s)
* 08:21 cscott@deploy1003: cscott: Continuing with deployment
* 08:16 cscott@deploy1003: cscott: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:14 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]]
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: Migration of db1244.eqiad.wmnet completed
* 08:02 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:02 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:01 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] (duration: 18m 58s)
* 08:01 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:01 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 07:59 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:59 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2006.wikimedia.org
* 07:58 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:57 cscott@deploy1003: cscott: Continuing with deployment
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:54 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2006.wikimedia.org
* 07:54 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:54 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:44 cscott@deploy1003: cscott: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:44 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 07:44 moritzm: installing node-lodash security updates
* 07:42 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]]
* 07:39 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 07:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] (duration: 07m 28s)
* 07:26 cscott@deploy1003: ssastry, cscott: Continuing with deployment
* 07:25 cscott@deploy1003: ssastry, cscott: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:23 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1244: Migration of db1244.eqiad.wmnet completed
* 07:22 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]]
* 07:16 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] (duration: 06m 55s)
* 07:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1244.eqiad.wmnet with OS trixie
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:09 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]]
* 06:54 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:50 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1250.eqiad.wmnet with OS trixie
* 06:34 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1244.eqiad.wmnet with OS trixie
* 06:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: dbmaint on s4@eqiad [[phab:T429893|T429893]]
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:14 cwilliams@dns1006: END - running authdns-update
* 06:12 cwilliams@dns1006: START - running authdns-update
* 06:11 cwilliams@dns1006: END - running authdns-update
* 06:11 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1244 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94676 and previous config saved to /var/cache/conftool/dbconfig/20260702-061059-cwilliams.json
* 06:09 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:09 cwilliams@dns1006: START - running authdns-update
* 06:08 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0)  on VRTS host vrts1003.eqiad.wmnet
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1160 to s4 primary and set section read-write [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94675 and previous config saved to /var/cache/conftool/dbconfig/20260702-060746-cwilliams.json
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94674 and previous config saved to /var/cache/conftool/dbconfig/20260702-060704-cwilliams.json
* 06:06 cezmunsta: Starting s4 eqiad failover from db1244 to db1160 - [[phab:T430817|T430817]]
* 06:04 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade  on VRTS host vrts1003.eqiad.wmnet
* 05:59 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1160 with weight 0 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94673 and previous config saved to /var/cache/conftool/dbconfig/20260702-055927-cwilliams.json
* 05:59 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430817|T430817]]
* 05:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1250.eqiad.wmnet with OS trixie
* 05:44 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1250.eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 05:39 marostegui: Failover m3 (phabricator) from db1250 to db1228 - [[phab:T430158|T430158]]
* 05:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 04:45 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] (duration: 09m 08s)
* 04:41 tstarling@deploy1003: tstarling, reedy: Continuing with deployment
* 04:38 tstarling@deploy1003: tstarling, reedy: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:36 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]]
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 59s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:16 ryankemper: [[phab:T429844|T429844]] [opensearch] completed `cirrussearch2111` reimage; all codfw search clusters are green, all nodes now report `OpenSearch 2.19.5`, and the temporary chi voting exclusion has been removed
* 00:57 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:29 ryankemper: [[phab:T429844|T429844]] [opensearch] depooled codfw search-omega/search-psi discovery records to match existing codfw search depool during OpenSearch 2.19 migration
* 00:29 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 00:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:01 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:00 ryankemper: [[phab:T429844|T429844]] [opensearch] chi cluster recovered after stopping `opensearch_1@production-search-codfw` on `cirrussearch2111`

== 2026-07-01 ==
* 23:59 ryankemper: [[phab:T429844|T429844]] [opensearch] stopped `opensearch_1@production-search-codfw` on `cirrussearch2111` after chi cluster-manager election churn following `voting_config_exclusions` POST; hoping this triggers a re-election
* 23:52 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 23:50 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2003.codfw.wmnet with OS bookworm
* 22:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 22:13 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2084.codfw.wmnet with OS trixie
* 22:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 22:03 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:01 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 21:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 21:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:36 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:35 bking@cumin2003: START - Cookbook sre.dns.netbox
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2003
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2003.codfw.wmnet with OS bookworm
* 21:19 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2084.codfw.wmnet with OS trixie
* 21:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2108.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:45 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:28 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 20:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 19:59 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2108.codfw.wmnet with OS trixie
* 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2093.codfw.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:43 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2080.codfw.wmnet with OS trixie
* 19:28 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:18 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:17 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 19:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 18:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2093.codfw.wmnet with OS trixie
* 18:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2080.codfw.wmnet with OS trixie
* 18:27 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 18:18 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] (duration: 09m 15s)
* 18:13 jgiannelos@deploy1003: jgiannelos, neriah: Continuing with deployment
* 18:11 jgiannelos@deploy1003: jgiannelos, neriah: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:09 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]]
* 17:40 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 16:58 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 30 hosts
* 16:57 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 30 hosts
* 16:52 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2202.codfw.wmnet
* 16:52 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2202.codfw.wmnet
* 16:51 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2012.codfw.wmnet
* 16:51 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs2012.codfw.wmnet
* 16:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2076.codfw.wmnet with OS trixie
* 16:49 brett: Start pybal on lvs2012 - [[phab:T429861|T429861]]
* 16:49 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.remove-downtime (exit_code=97) for 59 hosts
* 16:48 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 59 hosts
* 16:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2061.codfw.wmnet with OS trixie
* 16:30 dancy@deploy1003: Installation of scap version &quot;4.271.0&quot; completed for 2 hosts
* 16:28 dancy@deploy1003: Installing scap version &quot;4.271.0&quot; for 2 host(s)
* 16:23 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:18 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:18 jasmine@dns1004: END - running authdns-update
* 16:16 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:15 jasmine@dns1004: START - running authdns-update
* 16:14 jasmine@dns1004: END - running authdns-update
* 16:12 jasmine@dns1004: START - running authdns-update
* 16:07 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: maintenance
* 16:06 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt with reason: Junos upograde
* 16:00 papaul: ongoing maintenance on lsw1-b2-codfw
* 16:00 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2076.codfw.wmnet with OS trixie
* 15:59 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:59 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:57 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2061.codfw.wmnet with OS trixie
* 15:55 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:55 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:51 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:51 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Repooling after switchover
* 15:50 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:50 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 15:48 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2092.codfw.wmnet with OS trixie
* 15:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:30 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 15:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:22 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2012.codfw.wmnet with reason: Rack B2 maintenance - [[phab:T429861|T429861]]
* 15:21 brett: Stopping pybal on lvs2012 in preparation for codfw rack b2 maintenance - [[phab:T429861|T429861]]
* 15:20 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:12 _joe_: restarted manually alertmanager-irc-relay
* 15:12 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt with reason: Junos upograde
* 15:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 15:07 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:06 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:02 papaul: ongoing maintenance on lsw1-a8-codfw
* 14:31 topranks: POWERING DOWN CR1-EQIAD for line card installation [[phab:T426343|T426343]]
* 14:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] (duration: 08m 57s)
* 14:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
* 14:24 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:22 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]]
* 14:22 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:16 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:15 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:14 topranks: re-enable routing-engine graceful-failover on cr1-eqiad [[phab:T417873|T417873]]
* 14:13 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:13 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2220: Repooling after switchover
* 14:12 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:11 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:08 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] (duration: 10m 01s)
* 14:07 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2220 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94664 and previous config saved to /var/cache/conftool/dbconfig/20260701-140729-fceratto.json
* 14:06 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2159 to s7 primary [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94663 and previous config saved to /var/cache/conftool/dbconfig/20260701-140503-fceratto.json
* 14:04 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:04 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 14:04 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 14:04 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:04 dreamyjazz@deploy1003: anzx, dreamyjazz: Continuing with deployment
* 14:04 federico3: Starting s7 codfw failover from db2220 to db2159 - [[phab:T430826|T430826]]
* 14:03 jmm@dns1004: END - running authdns-update
* 14:03 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:03 topranks: flipping cr1-eqiad active routing-enginer back to RE0 [[phab:T417873|T417873]]
* 14:03 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudsw1-c8-eqiad,cloudsw1-d5-eqiad with reason: router upgrades eqiad
* 14:01 jmm@dns1004: START - running authdns-update
* 14:00 dreamyjazz@deploy1003: anzx, dreamyjazz: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2159 with weight 0 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94662 and previous config saved to /var/cache/conftool/dbconfig/20260701-135906-fceratto.json
* 13:58 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]]
* 13:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 [[phab:T430826|T430826]]
* 13:56 topranks: reboot routing-enginer RE0 on cr1-eqiad [[phab:T417873|T417873]]
* 13:48 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1006.wikimedia.org
* 13:44 atsuko@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:43 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1006.wikimedia.org
* 13:41 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1005.wikimedia.org
* 13:37 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1005.wikimedia.org
* 13:37 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on pfw1-eqiad with reason: router upgrades eqiad
* 13:35 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs[1017-1020].eqiad.wmnet with reason: router upgrades eqiad
* 13:34 caro@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] (duration: 07m 59s)
* 13:30 caro@deploy1003: caro: Continuing with deployment
* 13:28 caro@deploy1003: caro: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:27 topranks: route-engine failover cr1-eqiad
* 13:26 caro@deploy1003: Started scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]]
* 13:15 topranks: rebooting routing-engine 1 on cr1-eqiad [[phab:T417873|T417873]]
* 13:13 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] (duration: 08m 29s)
* 13:13 moritzm: installing qemu security updates
* 13:11 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 13:11 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 13:09 jgiannelos@deploy1003: jgiannelos: Continuing with deployment
* 13:08 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:07 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:06 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance
* 13:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2214: Repooling after switchover
* 13:05 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]]
* 13:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling after switchover
* 13:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2214 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94660 and previous config saved to /var/cache/conftool/dbconfig/20260701-130413-fceratto.json
* 13:01 moritzm: installing python3.13 security updates
* 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2229 to s6 primary [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94659 and previous config saved to /var/cache/conftool/dbconfig/20260701-125959-fceratto.json
* 12:59 federico3: Starting s6 codfw failover from db2214 to db2229 - [[phab:T430814|T430814]]
* 12:57 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 13 hosts with reason: router upgrade and line card install
* 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2229 with weight 0 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94658 and previous config saved to /var/cache/conftool/dbconfig/20260701-125149-fceratto.json
* 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Primary switchover s6 [[phab:T430814|T430814]]
* 12:50 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet
* 12:50 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet
* 12:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2100.codfw.wmnet with OS trixie
* 12:38 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2083.codfw.wmnet with OS trixie
* 12:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2240: Migration of db2240.codfw.wmnet completed
* 12:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:00 topranks: drain traffic on cr1-eqiad to allow for line card install and JunOS upgrade [[phab:T426343|T426343]]
* 11:52 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2083.codfw.wmnet with OS trixie
* 11:50 cmooney@dns2005: END - running authdns-update
* 11:49 cmooney@dns2005: START - running authdns-update
* 11:48 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2100.codfw.wmnet with OS trixie
* 11:40 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 11:40 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 11:31 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2240: Migration of db2240.codfw.wmnet completed
* 11:30 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 11:28 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
* 11:27 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:23 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2240.codfw.wmnet with OS trixie
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:17 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:16 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 11:16 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 11:15 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2086.codfw.wmnet with OS trixie
* 11:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2106.codfw.wmnet with OS trixie
* 11:14 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:13 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:12 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:09 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2115.codfw.wmnet with OS trixie
* 11:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 11:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 10:53 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:44 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:44 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2240.codfw.wmnet with OS trixie
* 10:44 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:42 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: dbmaint on s4@codfw [[phab:T429893|T429893]]
* 10:40 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 10:39 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:27 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2240 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94653 and previous config saved to /var/cache/conftool/dbconfig/20260701-102658-cwilliams.json
* 10:26 moritzm: installing nginx security updates
* 10:26 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2086.codfw.wmnet with OS trixie
* 10:23 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2179 to s4 primary [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94652 and previous config saved to /var/cache/conftool/dbconfig/20260701-102356-cwilliams.json
* 10:23 cezmunsta: Starting s4 codfw failover from db2240 to db2179 - [[phab:T430127|T430127]]
* 10:23 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2106.codfw.wmnet with OS trixie
* 10:20 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2115.codfw.wmnet with OS trixie
* 10:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2179 with weight 0 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94651 and previous config saved to /var/cache/conftool/dbconfig/20260701-101531-cwilliams.json
* 10:15 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430127|T430127]]
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:51 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 09:39 mszwarc@deploy1003: Synchronized private/SuggestedInvestigationsSignals/SuggestedInvestigationsSignal4n.php: Update SI signal 4n (duration: 06m 08s)
* 09:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:21 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:54 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:38 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:38 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:36 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:21 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] (duration: 36m 11s)
* 08:15 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:09 mszwarc@deploy1003: mszwarc, abi: Continuing with deployment
* 08:03 mszwarc@deploy1003: mszwarc, abi: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:55 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 07:51 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 07:45 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]]
* 07:30 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050] (duration: 00m 22s)
* 07:29 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050]
* 07:28 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050] (duration: 01m 59s)
* 07:28 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] (duration: 07m 19s)
* 07:26 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050]
* 07:26 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050] (duration: 04m 32s)
* 07:24 mszwarc@deploy1003: wmde-fisch, mszwarc: Continuing with deployment
* 07:23 mszwarc@deploy1003: wmde-fisch, mszwarc: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:21 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050]
* 07:21 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050] (duration: 02m 01s)
* 07:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]]
* 07:19 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050]
* 07:13 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] (duration: 09m 13s)
* 07:09 mszwarc@deploy1003: mszwarc, chlod, revi: Continuing with deployment
* 07:06 mszwarc@deploy1003: mszwarc, chlod, revi: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:04 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]]
* 06:55 elukey: upgrade all trixie hosts to pywmflib 3.0 - [[phab:T430552|T430552]]
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:31 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Niharika29 out of all services on: 2453 hosts
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.hiddenparma (exit_code=99) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:01 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1039.eqiad.wmnet with reason: issues
* 05:41 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1027.eqiad.wmnet
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2068.codfw.wmnet with OS trixie
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s2
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s7
* 05:36 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:16 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:15 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:09 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2067.codfw.wmnet with OS trixie
* 04:56 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2068.codfw.wmnet with OS trixie
* 04:49 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:45 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:27 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2067.codfw.wmnet with OS trixie
* 03:47 slyngshede@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Hardware crash
* 03:21 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:59 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:55 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:35 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:31 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:30 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:26 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:09 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 54s)
* 02:03 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:07 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad back to read-write - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94649 and previous config saved to /var/cache/conftool/dbconfig/20260701-010716-ladsgroup.json
* 01:05 ladsgroup@dns1004: END - running authdns-update
* 01:05 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool es1039 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94648 and previous config saved to /var/cache/conftool/dbconfig/20260701-010551-ladsgroup.json
* 01:03 ladsgroup@dns1004: START - running authdns-update
* 01:00 ladsgroup@cumin1003: dbctl commit (dc=all): 'Promote es1035 to es7 primary [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94647 and previous config saved to /var/cache/conftool/dbconfig/20260701-010002-ladsgroup.json
* 00:58 Amir1: Starting es7 eqiad failover from es1039 to es1035 - [[phab:T430765|T430765]]
* 00:53 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es1035 with weight 0 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94646 and previous config saved to /var/cache/conftool/dbconfig/20260701-005329-ladsgroup.json
* 00:53 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Primary switchover es7 [[phab:T430765|T430765]]
* 00:42 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad as read-only for maintenance - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94645 and previous config saved to /var/cache/conftool/dbconfig/20260701-004221-ladsgroup.json
* 00:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2102.codfw.wmnet with OS trixie
* 00:15 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2103.codfw.wmnet with OS trixie
* 00:05 dr0ptp4kt: DEPLOYED Refinery at {{Gerrit|4e7a2b32}} for changes: pageview allowlist {{Gerrit|1305158}} (+min.wikiquote) {{Gerrit|1305162}} (+bol.wikipedia), {{Gerrit|1305156}} (+isv.wikipedia); {{Gerrit|1305980}} (pv allowlist -api.wikimedia, sqoop +isvwiki); sqoop {{Gerrit|1295064}} (+globalimagelinks) {{Gerrit|1295069}} (+filerevision) using scap, then deployed onto HDFS (manual copyToLocal required additionally)

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>jf9gi35rmcqt76geo5bsqxeevnqwkqt</sha1>
    </revision>
    <revision>
      <id>2433180</id>
      <parentid>2433179</parentid>
      <timestamp>2026-07-05T02:01:30Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 08s)</comment>
      <origin>2433180</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="108371" sha1="h1g9s6ypgebuotr3j7e2naizksav1bg" xml:space="preserve">
== 2026-07-05 ==
* 02:01 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 01m 08s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-04 ==
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 58s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-03 ==
* 17:08 topranks: revert protocol preference changes on cr3-ulsfo after upgrade
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr2-eqord with reason: upgrade JunOS cr3-ulsfo
* 16:53 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr4-ulsfo with reason: upgrade JunOS cr3-ulsfo
* 16:48 topranks: reboot cr3-ulsfo to upgrade JunOS and reset linecard [[phab:T424839|T424839]]
* 15:52 topranks: adjust outbound BGP policies on cr3-ulsfo to drain router of traffic [[phab:T424839|T424839]]
* 15:45 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on lvs[4008-4010].ulsfo.wmnet with reason: upgrade JunOS cr3-ulsfo
* 15:44 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on asw1-[22-23]-ulsfo,cr3-ulsfo,cr3-ulsfo IPv6,cr3-ulsfo.mgmt with reason: upgrade JunOS cr3-ulsfo
* 15:36 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 15:35 atsuko@cumin1003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 14:40 cmooney@dns3003: END - running authdns-update
* 14:26 cmooney@dns3003: START - running authdns-update
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 14:26 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:19 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to ulsfo - cmooney@cumin1003&quot;
* 14:16 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 13:38 sukhe@dns1004: END - running authdns-update
* 13:35 sukhe@dns1004: START - running authdns-update
* 13:26 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:26 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:26 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:24 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:24 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:18 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 13:17 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest1005.eqiad.wmnet
* 13:16 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 13:16 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:15 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:14 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:14 moritzm: imported samplicator 1.3.8rc1-1+deb13u1 to trixie-wikimedia/main [[phab:T337208|T337208]]
* 13:13 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:07 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 13:07 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 13:02 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=99) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:02 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 13:00 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:58 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:57 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:57 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:53 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:52 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest[2001,2003-2004,2006,2009-2010].codfw.wmnet,sretest[1005-1006].eqiad.wmnet
* 12:50 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for host sretest1005.eqiad.wmnet
* 12:50 elukey@cumin1003: START - Cookbook sre.hosts.bmc-user-mgmt for host sretest1005.eqiad.wmnet
* 12:47 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:41 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:40 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
* 12:39 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
* 12:32 kamila@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
* 12:26 kamila@cumin1003: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
* 12:23 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 12:19 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[2004-2007].codfw.wmnet
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:15 jynus@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:15 jynus@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[2004-2007].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2003&quot;
* 12:09 jynus@cumin2003: START - Cookbook sre.dns.netbox
* 11:58 jynus@cumin2003: START - Cookbook sre.hosts.decommission for hosts backup[2004-2007].codfw.wmnet
* 10:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup[1004-1007].eqiad.wmnet
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:02 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 10:01 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: backup[1004-1007].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003&quot;
* 09:52 jynus@cumin1003: START - Cookbook sre.dns.netbox
* 09:39 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:36 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts backup[1004-1007].eqiad.wmnet
* 09:36 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 09:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:16 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 09:05 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:04 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 09:00 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:59 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:57 dpogorzelski@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 08:55 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 08:50 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 08:49 atsuko@cumin1003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 08:49 atsukoito: depooling cirrussearch in codfw because of regression after upgrade [[phab:T431091|T431091]]
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mirror1001.wikimedia.org
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 08:31 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:29 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: mirror1001.wikimedia.org decommissioned, removing all IPs except the asset tag one - jmm@cumin2003&quot;
* 08:18 jmm@cumin2003: START - Cookbook sre.dns.netbox
* 08:11 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts mirror1001.wikimedia.org
* 06:15 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 18s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

== 2026-07-02 ==
* 22:55 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1003.wikimedia.org with OS trixie
* 22:29 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:23 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1003.wikimedia.org with reason: host reimage
* 22:05 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint1003.wikimedia.org with OS trixie
* 22:03 mutante: contint1003 (zuul.wikimedia.org) - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 22:03 dzahn@cumin2002: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on zuul.wikimedia.org with reason: reimage
* 21:39 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 18s)
* 21:39 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 21:20 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 21:19 sbassett: Deployed security fix for [[phab:T428829|T428829]]
* 20:58 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:55 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin[2002-2003].codfw.wmnet,cumin1003.eqiad.wmnet with reason: Release v0.11.2 update for new Aerleon - cmooney@cumin1003
* 20:40 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] (duration: 12m 35s)
* 20:36 arlolra@deploy1003: cscott, arlolra: Continuing with deployment
* 20:35 bking@deploy1003: Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]] (duration: 00m 20s)
* 20:35 bking@deploy1003: Started deploy [wdqs/wdqs@e8fb00c] (wcqs): [[phab:T430879|T430879]]
* 20:33 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint2003.wikimedia.org with OS trixie
* 20:31 arlolra@deploy1003: cscott, arlolra: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Cha
* 20:28 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1307218{{!}}Revert &quot;Temporarily disable experimental ExtTagPFragment type&quot; (T430344 T429624)]], [[gerrit:1307227{{!}}Preview: Ensure ParserMigration's handler is called to setUseParsoid (T429408)]], [[gerrit:1307223{{!}}Ensure ParserMigration is consulted if Parsoid should be used (T429408)]]
* 20:17 sbassett@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] (duration: 08m 13s)
* 20:14 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:13 sbassett@deploy1003: sbassett: Continuing with deployment
* 20:11 sbassett@deploy1003: sbassett: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 20:09 sbassett@deploy1003: Started scap sync-world: Backport for [[gerrit:1307196{{!}}mediawiki.action.edit.preview: Fix compat with `&lt;button&gt;`-buttons (T430956)]]
* 20:08 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 20:08 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 20:08 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint2003.wikimedia.org with reason: host reimage
* 20:05 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs1002.eqiad.wmnet -&gt; wcqs1003.eqiad.wmnet, repooling source-only afterwards
* 19:49 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint2003.wikimedia.org with OS trixie
* 19:48 mutante: contint2003 - reimaging because of [[phab:T430510|T430510]]#12067628  [[phab:T418521|T418521]]
* 18:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 18:13 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 17:58 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1003.eqiad.wmnet with OS bookworm
* 17:52 jasmine@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:52 jasmine@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
* 17:51 jasmine@cumin2002: conftool action : set/pooled=yes:weight=10; selector: name=wikikube-ctrl1005.eqiad.wmnet
* 17:48 jasmine_: homer &quot;cr*eqiad*&quot; commit &quot;Added new stacked control plane wikikube-ctrl1005&quot;
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
* 17:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
* 17:31 ladsgroup@deploy1003: Finished scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] (duration: 09m 33s)
* 17:26 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
* 17:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:23 ladsgroup@deploy1003: ladsgroup: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 17:21 ladsgroup@deploy1003: Started scap sync-world: Backport for [[gerrit:1292300{{!}}etcd: Ignore test-s4 from dbctl (T427059)]]
* 17:18 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1003.eqiad.wmnet with reason: host reimage
* 17:16 rscout@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
* 17:16 rscout@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
* 17:15 rscout@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
* 17:12 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on wcqs[2002-2003].codfw.wmnet,wcqs1002.eqiad.wmnet with reason: reimaging hosts
* 17:08 bd808@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
* 17:08 bd808@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
* 17:07 bd808@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
* 17:05 bd808@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
* 17:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:03 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:03 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;running to make sure all updates are synced - cmooney@cumin1003&quot;
* 17:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs1003
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs1003
* 17:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 17:00 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs1003.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Re-running - btullis@cumin1003&quot;
* 16:58 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer ([[phab:T430879|T430879]], restore data on newly-reimaged host) xfer commons from wcqs2002.codfw.wmnet -&gt; wcqs2003.codfw.wmnet, repooling source-only afterwards
* 16:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:58 btullis@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:57 tappof: bump space for prometheus k8s-aux in eqiad
* 16:55 cmooney@dns3003: END - running authdns-update
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:55 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to eqsin - cmooney@cumin1003&quot;
* 16:53 cmooney@dns3003: START - running authdns-update
* 16:52 ryankemper: [ml-serve-eqiad] Cleared out 1302 failed (Evicted) pods: `kubectl -n llm delete pods --field-selector=status.phase=Failed`, freeing calico-kube-controllers from OOM crashloop (evictions were caused by disk pressure)
* 16:49 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1003&quot;
* 16:46 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 16:39 rzl@dns1004: END - running authdns-update
* 16:37 rzl@dns1004: START - running authdns-update
* 16:36 rzl@dns1004: START - running authdns-update
* 16:35 rzl@deploy1003: Finished scap sync-world: [[phab:T416623|T416623]] (duration: 10m 19s)
* 16:34 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:33 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:30 rzl@deploy1003: rzl: Continuing with deployment
* 16:28 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1004.eqiad.wmnet with reason: host reimage
* 16:26 rzl@deploy1003: rzl: [[phab:T416623|T416623]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 16:25 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:25 rzl@deploy1003: Started scap sync-world: [[phab:T416623|T416623]]
* 16:25 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: sync
* 16:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: sync
* 16:16 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1004.eqiad.wmnet with OS bookworm
* 16:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-master1003.eqiad.wmnet with OS bookworm
* 16:11 cwilliams@cumin1003: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test[2001-2002].codfw.wmnet
* 16:11 cwilliams@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Security updates
* 16:08 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 16:08 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 16:08 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1023: Security updates
* 15:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 15:54 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-master1003.eqiad.wmnet with reason: host reimage
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Security updates
* 15:45 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:45 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:45 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1023: Security updates
* 15:42 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-master1003.eqiad.wmnet with OS bookworm
* 15:24 moritzm: installing busybox updates from bookworm point release
* 15:20 moritzm: installing busybox updates from trixie point release
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Security updates
* 15:15 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 15:15 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 15:15 root@cumin1003: START - Cookbook sre.mysql.pool pool pc1021: Security updates
* 15:13 moritzm: installing giflib security updates
* 15:08 moritzm: installing Tomcat security updates
* 14:57 atsuko@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
* 14:56 atsuko@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
* 14:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:53 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:53 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Security updates
* 14:53 root@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
* 14:53 root@cumin1003: START - Cookbook sre.mysql.parsercache
* 14:53 root@cumin1003: START - Cookbook sre.mysql.depool depool pc1021: Security updates
* 14:53 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Unblock taavi - oblivian@cumin1003
* 14:52 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Unblock taavi - oblivian@cumin1003&quot;
* 14:46 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94711 and previous config saved to /var/cache/conftool/dbconfig/20260702-144644-fceratto.json
* 14:36 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94709 and previous config saved to /var/cache/conftool/dbconfig/20260702-143636-fceratto.json
* 14:32 moritzm: installing libdbi-perl security updates
* 14:26 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P94708 and previous config saved to /var/cache/conftool/dbconfig/20260702-142628-fceratto.json
* 14:16 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94707 and previous config saved to /var/cache/conftool/dbconfig/20260702-141621-fceratto.json
* 14:12 moritzm: installing rsync security updates
* 14:11 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 14:10 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2205 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94706 and previous config saved to /var/cache/conftool/dbconfig/20260702-140959-fceratto.json
* 14:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2205.codfw.wmnet with reason: Maintenance
* 14:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 14:07 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 14:06 Tran: Deployed patch for [[phab:T427287|T427287]]
* 14:04 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:59 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:59 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2205: Repooling after switchover
* 13:59 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:55 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2205: Repooling after switchover
* 13:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2205 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94704 and previous config saved to /var/cache/conftool/dbconfig/20260702-135505-fceratto.json
* 13:54 moritzm: installing sed security updates
* 13:53 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:52 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2209 to s3 primary [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94703 and previous config saved to /var/cache/conftool/dbconfig/20260702-135235-fceratto.json
* 13:52 federico3: Starting s3 codfw failover from db2205 to db2209 - [[phab:T430912|T430912]]
* 13:51 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:51 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:48 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:47 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2209 with weight 0 [[phab:T430912|T430912]]', diff saved to https://phabricator.wikimedia.org/P94702 and previous config saved to /var/cache/conftool/dbconfig/20260702-134719-fceratto.json
* 13:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 [[phab:T430912|T430912]]
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
* 13:44 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
* 13:40 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 13:37 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search,name=codfw
* 13:36 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:36 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:34 bking@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 13:30 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:29 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:27 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:26 elukey@cumin1003: START - Cookbook sre.hosts.provision for host an-test-master1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
* 13:25 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
* 13:23 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-psi,name=codfw
* 13:22 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search-omega,name=codfw
* 13:17 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=search,name=codfw
* 13:17 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns1004.wikimedia.org
* 13:12 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart (exit_code=97) rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox and (A:dnsbox)
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=97) rolling restart_daemons on A:wikidough
* 13:11 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
* 13:09 aude@deploy1003: Finished scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] (duration: 07m 20s)
* 13:05 aude@deploy1003: jdrewniak, aude: Continuing with deployment
* 13:04 aude@deploy1003: jdrewniak, aude: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:02 aude@deploy1003: Started scap sync-world: Backport for [[gerrit:1305773{{!}}Phase 3 Legal contact link deployments. (T430227)]]
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs-categories1001.eqiad.wmnet
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 12:19 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 12:10 jmm@dns1004: END - running authdns-update
* 12:07 jmm@dns1004: START - running authdns-update
* 11:51 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: wdqs-categories1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003&quot;
* 11:44 btullis@cumin1003: START - Cookbook sre.dns.netbox
* 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:42 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 11:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver1003.eqiad.wmnet
* 11:39 btullis@cumin1003: START - Cookbook sre.hosts.decommission for hosts wdqs-categories1001.eqiad.wmnet
* 11:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver1003.eqiad.wmnet
* 11:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetserver2004.codfw.wmnet
* 11:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host puppetserver2004.codfw.wmnet
* 11:29 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.disable-merges (exit_code=0)
* 11:29 jmm@cumin2003: START - Cookbook sre.puppet.disable-merges
* 10:57 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2214: Repooling
* 10:49 jmm@dns1004: END - running authdns-update
* 10:47 jmm@dns1004: START - running authdns-update
* 10:31 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94698 and previous config saved to /var/cache/conftool/dbconfig/20260702-103146-fceratto.json
* 10:21 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94696 and previous config saved to /var/cache/conftool/dbconfig/20260702-102137-fceratto.json
* 10:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:19 fnegri@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb1017.eqiad.wmnet
* 10:18 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
* 10:18 fceratto@cumin1003: Removing es1033 from zarcillo [[phab:T408772|T408772]]
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1033.eqiad.wmnet
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 10:14 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:14 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: es1033.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003&quot;
* 10:13 fnegri@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1017.eqiad.wmnet
* 10:12 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2214.codfw.wmnet
* 10:12 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling
* 10:11 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P94693 and previous config saved to /var/cache/conftool/dbconfig/20260702-101130-fceratto.json
* 10:10 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:10 fceratto@cumin1003: START - Cookbook sre.dns.netbox
* 10:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 10:03 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts es1033.eqiad.wmnet
* 10:03 fceratto@cumin1003: START - Cookbook sre.mysql.decommission
* 10:01 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94691 and previous config saved to /var/cache/conftool/dbconfig/20260702-100122-fceratto.json
* 09:55 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2213 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94690 and previous config saved to /var/cache/conftool/dbconfig/20260702-095529-fceratto.json
* 09:55 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: Maintenance
* 09:54 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:53 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2213: Repooling after switchover
* 09:51 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:44 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2213: Repooling after switchover
* 09:39 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2213 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94688 and previous config saved to /var/cache/conftool/dbconfig/20260702-093859-fceratto.json
* 09:36 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2192 to s5 primary [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94687 and previous config saved to /var/cache/conftool/dbconfig/20260702-093650-fceratto.json
* 09:36 federico3: Starting s5 codfw failover from db2213 to db2192 - [[phab:T430923|T430923]]
* 09:30 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94686 and previous config saved to /var/cache/conftool/dbconfig/20260702-093004-fceratto.json
* 09:24 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2192 with weight 0 [[phab:T430923|T430923]]', diff saved to https://phabricator.wikimedia.org/P94685 and previous config saved to /var/cache/conftool/dbconfig/20260702-092455-fceratto.json
* 09:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 23 hosts with reason: Primary switchover s5 [[phab:T430923|T430923]]
* 09:19 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94684 and previous config saved to /var/cache/conftool/dbconfig/20260702-091957-fceratto.json
* 09:16 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] (duration: 06m 57s)
* 09:13 moritzm: installing libgcrypt20 security updates
* 09:12 kharlan@deploy1003: kharlan: Continuing with deployment
* 09:11 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 09:09 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P94683 and previous config saved to /var/cache/conftool/dbconfig/20260702-090950-fceratto.json
* 09:09 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307076{{!}}SourceEditorOverlay: Re-enable buttons after non-captcha save failure (T430518)]]
* 09:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 09:01 kharlan@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] (duration: 07m 07s)
* 08:59 fceratto@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94682 and previous config saved to /var/cache/conftool/dbconfig/20260702-085942-fceratto.json
* 08:57 kharlan@deploy1003: kharlan: Continuing with deployment
* 08:56 kharlan@deploy1003: kharlan: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:54 kharlan@deploy1003: Started scap sync-world: Backport for [[gerrit:1307075{{!}}build: Update required Node version from 24.14.1 to 24.18.0]]
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:52 fceratto@cumin1003: dbctl commit (dc=all): 'Depooling db2220 ([[phab:T426633|T426633]])', diff saved to https://phabricator.wikimedia.org/P94681 and previous config saved to /var/cache/conftool/dbconfig/20260702-085237-fceratto.json
* 08:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
* 08:43 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 08:40 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:25 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] (duration: 11m 44s)
* 08:21 cscott@deploy1003: cscott: Continuing with deployment
* 08:16 cscott@deploy1003: cscott: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 08:14 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307059{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T387374 T430186 T430367 T430501)]], [[gerrit:1307061{{!}}Bump wikimedia/parsoid to 0.24.0-a14 (T430501)]]
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 08:08 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: Migration of db1244.eqiad.wmnet completed
* 08:02 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:02 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
* 08:01 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] (duration: 18m 58s)
* 08:01 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:01 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 08:00 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
* 07:59 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:59 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2006.wikimedia.org
* 07:58 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:57 cscott@deploy1003: cscott: Continuing with deployment
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:56 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:55 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
* 07:54 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2006.wikimedia.org
* 07:54 ozge@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:54 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 07:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply
* 07:44 cscott@deploy1003: cscott: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:44 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader2005.wikimedia.org
* 07:44 moritzm: installing node-lodash security updates
* 07:42 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1307058{{!}}[REST] Don't language-convert non-parsoid output; don't lookup bogus titles (T430778)]], [[gerrit:1306996{{!}}[parser] When expanding an extension tag with a title, use a new frame (T430344 T429624)]]
* 07:39 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader2005.wikimedia.org
* 07:30 cscott@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] (duration: 07m 28s)
* 07:26 cscott@deploy1003: ssastry, cscott: Continuing with deployment
* 07:25 cscott@deploy1003: ssastry, cscott: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:23 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1244: Migration of db1244.eqiad.wmnet completed
* 07:22 cscott@deploy1003: Started scap sync-world: Backport for [[gerrit:1306985{{!}}Parsoid read views: Bump enwiki NS_MAIN desktop traffic to 100% (T430194)]]
* 07:16 wmde-fisch@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] (duration: 06m 55s)
* 07:13 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1244.eqiad.wmnet with OS trixie
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Continuing with deployment
* 07:11 wmde-fisch@deploy1003: wmde-fisch: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:09 wmde-fisch@deploy1003: Started scap sync-world: Backport for [[gerrit:1306970{{!}}Fix how to check the treatment group (T415904)]], [[gerrit:1306971{{!}}Fix how to check the treatment group (T415904)]]
* 06:54 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:50 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1244.eqiad.wmnet with reason: host reimage
* 06:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1250.eqiad.wmnet with OS trixie
* 06:34 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db1244.eqiad.wmnet with OS trixie
* 06:25 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1244: Upgrading db1244.eqiad.wmnet
* 06:25 cwilliams@cumin1003: dbmaint on s4@eqiad [[phab:T429893|T429893]]
* 06:25 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 06:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:14 cwilliams@dns1006: END - running authdns-update
* 06:12 cwilliams@dns1006: START - running authdns-update
* 06:11 cwilliams@dns1006: END - running authdns-update
* 06:11 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db1244 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94676 and previous config saved to /var/cache/conftool/dbconfig/20260702-061059-cwilliams.json
* 06:09 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1250.eqiad.wmnet with reason: host reimage
* 06:09 cwilliams@dns1006: START - running authdns-update
* 06:08 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0)  on VRTS host vrts1003.eqiad.wmnet
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db1160 to s4 primary and set section read-write [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94675 and previous config saved to /var/cache/conftool/dbconfig/20260702-060746-cwilliams.json
* 06:07 cwilliams@cumin1003: dbctl commit (dc=all): 'Set s4 eqiad as read-only for maintenance - [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94674 and previous config saved to /var/cache/conftool/dbconfig/20260702-060704-cwilliams.json
* 06:06 cezmunsta: Starting s4 eqiad failover from db1244 to db1160 - [[phab:T430817|T430817]]
* 06:04 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade  on VRTS host vrts1003.eqiad.wmnet
* 05:59 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db1160 with weight 0 [[phab:T430817|T430817]]', diff saved to https://phabricator.wikimedia.org/P94673 and previous config saved to /var/cache/conftool/dbconfig/20260702-055927-cwilliams.json
* 05:59 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430817|T430817]]
* 05:55 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host db1250.eqiad.wmnet with OS trixie
* 05:44 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1250.eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 05:39 marostegui: Failover m3 (phabricator) from db1250 to db1228 - [[phab:T430158|T430158]]
* 05:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2234].codfw.wmnet,db[1217,1228,1250].eqiad.wmnet with reason: m3 master switchover [[phab:T430158|T430158]]
* 04:45 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] (duration: 09m 08s)
* 04:41 tstarling@deploy1003: tstarling, reedy: Continuing with deployment
* 04:38 tstarling@deploy1003: tstarling, reedy: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 04:36 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1298928{{!}}CommonSettings: Set $wgScoreUseSvg = true (T49578)]]
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 59s)
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:16 ryankemper: [[phab:T429844|T429844]] [opensearch] completed `cirrussearch2111` reimage; all codfw search clusters are green, all nodes now report `OpenSearch 2.19.5`, and the temporary chi voting exclusion has been removed
* 00:57 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:29 ryankemper: [[phab:T429844|T429844]] [opensearch] depooled codfw search-omega/search-psi discovery records to match existing codfw search depool during OpenSearch 2.19 migration
* 00:29 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=codfw
* 00:29 ryankemper@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=codfw
* 00:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2111.codfw.wmnet with reason: host reimage
* 00:01 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2111.codfw.wmnet with OS trixie
* 00:00 ryankemper: [[phab:T429844|T429844]] [opensearch] chi cluster recovered after stopping `opensearch_1@production-search-codfw` on `cirrussearch2111`

== 2026-07-01 ==
* 23:59 ryankemper: [[phab:T429844|T429844]] [opensearch] stopped `opensearch_1@production-search-codfw` on `cirrussearch2111` after chi cluster-manager election churn following `voting_config_exclusions` POST; hoping this triggers a re-election
* 23:52 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
* 23:51 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
* 23:50 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
* 22:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2003.codfw.wmnet with OS bookworm
* 22:29 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 22:13 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2084.codfw.wmnet with OS trixie
* 22:09 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 22:03 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 22:01 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2003.codfw.wmnet with reason: host reimage
* 21:50 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 21:43 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2003
* 21:42 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wcqs2003
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wcqs2003.codfw.wmnet 45.48.192.10.in-addr.arpa 5.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 21:42 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2003 - bking@cumin2003&quot;
* 21:36 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2084.codfw.wmnet with reason: host reimage
* 21:35 bking@cumin2003: START - Cookbook sre.dns.netbox
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wcqs2003
* 21:34 bking@cumin2003: START - Cookbook sre.hosts.reimage for host wcqs2003.codfw.wmnet with OS bookworm
* 21:19 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2084.codfw.wmnet with OS trixie
* 21:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2108.codfw.wmnet with OS trixie
* 20:50 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:45 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2081.codfw.wmnet with reason: host reimage
* 20:28 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2081.codfw.wmnet with OS trixie
* 20:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 20:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2108.codfw.wmnet with reason: host reimage
* 19:59 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2108.codfw.wmnet with OS trixie
* 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2093.codfw.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 19:44 jasmine@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:43 jasmine@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.hosts.reimage: Host reimage - jasmine@cumin2002&quot;
* 19:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2080.codfw.wmnet with OS trixie
* 19:28 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:24 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:18 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl1005.eqiad.wmnet with reason: host reimage
* 19:17 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2093.codfw.wmnet with reason: host reimage
* 19:15 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 19:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2080.codfw.wmnet with reason: host reimage
* 18:57 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2093.codfw.wmnet with OS trixie
* 18:50 bking@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2080.codfw.wmnet with OS trixie
* 18:27 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 18:18 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] (duration: 09m 15s)
* 18:13 jgiannelos@deploy1003: jgiannelos, neriah: Continuing with deployment
* 18:11 jgiannelos@deploy1003: jgiannelos, neriah: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 18:09 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306950{{!}}PageBundleParserOutputConverter: Avoid revision lookup for bogus title (T430778)]], [[gerrit:1306910{{!}}PageBundleParserOutputConverter: Check for proper page before adding id/ns metadata (T430778)]]
* 17:40 jasmine@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-ctrl1005.eqiad.wmnet with OS trixie
* 16:58 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 30 hosts
* 16:57 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 30 hosts
* 16:52 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2202.codfw.wmnet
* 16:52 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2202.codfw.wmnet
* 16:51 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt
* 16:51 brett@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs2012.codfw.wmnet
* 16:51 brett@cumin2002: START - Cookbook sre.hosts.remove-downtime for lvs2012.codfw.wmnet
* 16:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2076.codfw.wmnet with OS trixie
* 16:49 brett: Start pybal on lvs2012 - [[phab:T429861|T429861]]
* 16:49 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.remove-downtime (exit_code=97) for 59 hosts
* 16:48 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for 59 hosts
* 16:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2061.codfw.wmnet with OS trixie
* 16:30 dancy@deploy1003: Installation of scap version &quot;4.271.0&quot; completed for 2 hosts
* 16:28 dancy@deploy1003: Installing scap version &quot;4.271.0&quot; for 2 host(s)
* 16:23 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:18 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2076.codfw.wmnet with reason: host reimage
* 16:18 jasmine@dns1004: END - running authdns-update
* 16:16 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host restbase2039.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
* 16:16 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2061.codfw.wmnet with reason: host reimage
* 16:15 jasmine@dns1004: START - running authdns-update
* 16:14 jasmine@dns1004: END - running authdns-update
* 16:12 jasmine@dns1004: START - running authdns-update
* 16:07 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: maintenance
* 16:06 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b2-codfw,lsw1-b2-codfw IPv6,lsw1-b2-codfw.mgmt with reason: Junos upograde
* 16:00 papaul: ongoing maintenance on lsw1-b2-codfw
* 16:00 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2076.codfw.wmnet with OS trixie
* 15:59 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:59 pt1979@cumin1003: START - Cookbook sre.hosts.remove-downtime for lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt
* 15:57 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2061.codfw.wmnet with OS trixie
* 15:55 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:55 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:51 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:51 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Repooling after switchover
* 15:50 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
* 15:50 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 15:48 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2092.codfw.wmnet with OS trixie
* 15:41 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:40 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
* 15:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
* 15:35 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
* 15:30 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
* 15:26 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
* 15:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:23 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:22 brett@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2012.codfw.wmnet with reason: Rack B2 maintenance - [[phab:T429861|T429861]]
* 15:21 brett: Stopping pybal on lvs2012 in preparation for codfw rack b2 maintenance - [[phab:T429861|T429861]]
* 15:20 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
* 15:12 _joe_: restarted manually alertmanager-irc-relay
* 15:12 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2092.codfw.wmnet with reason: host reimage
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
* 15:12 pt1979@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-a8-codfw,lsw1-a8-codfw IPv6,lsw1-a8-codfw.mgmt with reason: Junos upograde
* 15:09 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 15:07 pt1979@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:06 pt1979@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042,2046].codfw.wmnet
* 15:02 papaul: ongoing maintenance on lsw1-a8-codfw
* 14:31 topranks: POWERING DOWN CR1-EQIAD for line card installation [[phab:T426343|T426343]]
* 14:31 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] (duration: 08m 57s)
* 14:29 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
* 14:24 dreamyjazz@deploy1003: dreamyjazz: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 14:22 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306938{{!}}Remove group permissions definitions later in the request (T425048)]]
* 14:22 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:16 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:15 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2220: Repooling after switchover
* 14:14 topranks: re-enable routing-engine graceful-failover on cr1-eqiad [[phab:T417873|T417873]]
* 14:13 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:13 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2220: Repooling after switchover
* 14:12 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:12 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:11 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:10 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:08 dreamyjazz@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] (duration: 10m 01s)
* 14:07 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2220: Repooling after switchover
* 14:07 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2220 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94664 and previous config saved to /var/cache/conftool/dbconfig/20260701-140729-fceratto.json
* 14:06 jforrester@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
* 14:06 jforrester@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
* 14:05 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2159 to s7 primary [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94663 and previous config saved to /var/cache/conftool/dbconfig/20260701-140503-fceratto.json
* 14:04 jforrester@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
* 14:04 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 14:04 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 14:04 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
* 14:04 dreamyjazz@deploy1003: anzx, dreamyjazz: Continuing with deployment
* 14:04 federico3: Starting s7 codfw failover from db2220 to db2159 - [[phab:T430826|T430826]]
* 14:03 jmm@dns1004: END - running authdns-update
* 14:03 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
* 14:03 topranks: flipping cr1-eqiad active routing-enginer back to RE0 [[phab:T417873|T417873]]
* 14:03 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudsw1-c8-eqiad,cloudsw1-d5-eqiad with reason: router upgrades eqiad
* 14:01 jmm@dns1004: START - running authdns-update
* 14:00 dreamyjazz@deploy1003: anzx, dreamyjazz: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:59 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2159 with weight 0 [[phab:T430826|T430826]]', diff saved to https://phabricator.wikimedia.org/P94662 and previous config saved to /var/cache/conftool/dbconfig/20260701-135906-fceratto.json
* 13:58 dreamyjazz@deploy1003: Started scap sync-world: Backport for [[gerrit:1306456{{!}}eswikisource: add wikibooks as importsource (T430537)]], [[gerrit:1306916{{!}}Move non temporary accounts settings out TA section]], [[gerrit:1306925{{!}}Remove TA patrol rights from users on fishbowl + private (T425048)]]
* 13:57 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 [[phab:T430826|T430826]]
* 13:56 topranks: reboot routing-enginer RE0 on cr1-eqiad [[phab:T417873|T417873]]
* 13:48 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1006.wikimedia.org
* 13:44 atsuko@cumin2003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:43 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1006.wikimedia.org
* 13:41 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2092.codfw.wmnet with OS trixie
* 13:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM urldownloader1005.wikimedia.org
* 13:37 jmm@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM urldownloader1005.wikimedia.org
* 13:37 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on pfw1-eqiad with reason: router upgrades eqiad
* 13:35 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs[1017-1020].eqiad.wmnet with reason: router upgrades eqiad
* 13:34 caro@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] (duration: 07m 59s)
* 13:30 caro@deploy1003: caro: Continuing with deployment
* 13:28 caro@deploy1003: caro: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:27 topranks: route-engine failover cr1-eqiad
* 13:26 caro@deploy1003: Started scap sync-world: Backport for [[gerrit:1306842{{!}}EditCheck: fix pre-save focusedAction error (T430741)]]
* 13:15 topranks: rebooting routing-engine 1 on cr1-eqiad [[phab:T417873|T417873]]
* 13:13 jgiannelos@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] (duration: 08m 29s)
* 13:13 moritzm: installing qemu security updates
* 13:11 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 13:11 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 13:09 jgiannelos@deploy1003: jgiannelos: Continuing with deployment
* 13:08 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:07 jgiannelos@deploy1003: jgiannelos: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 13:06 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 13:06 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance
* 13:05 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2214: Repooling after switchover
* 13:05 jgiannelos@deploy1003: Started scap sync-world: Backport for [[gerrit:1306873{{!}}Parsoid read views: Bump enwiki traffic to 75%]]
* 13:04 fceratto@cumin1003: START - Cookbook sre.mysql.pool pool db2214: Repooling after switchover
* 13:04 fceratto@cumin1003: dbctl commit (dc=all): 'Depool db2214 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94660 and previous config saved to /var/cache/conftool/dbconfig/20260701-130413-fceratto.json
* 13:01 moritzm: installing python3.13 security updates
* 13:00 fceratto@cumin1003: dbctl commit (dc=all): 'Promote db2229 to s6 primary [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94659 and previous config saved to /var/cache/conftool/dbconfig/20260701-125959-fceratto.json
* 12:59 federico3: Starting s6 codfw failover from db2214 to db2229 - [[phab:T430814|T430814]]
* 12:57 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 13 hosts with reason: router upgrade and line card install
* 12:51 fceratto@cumin1003: dbctl commit (dc=all): 'Set db2229 with weight 0 [[phab:T430814|T430814]]', diff saved to https://phabricator.wikimedia.org/P94658 and previous config saved to /var/cache/conftool/dbconfig/20260701-125149-fceratto.json
* 12:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 21 hosts with reason: Primary switchover s6 [[phab:T430814|T430814]]
* 12:50 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2189.codfw.wmnet
* 12:50 fceratto@cumin1003: START - Cookbook sre.hosts.remove-downtime for db2189.codfw.wmnet
* 12:42 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2100.codfw.wmnet with OS trixie
* 12:38 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2083.codfw.wmnet with OS trixie
* 12:19 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.major-upgrade (exit_code=0)
* 12:17 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2240: Migration of db2240.codfw.wmnet completed
* 12:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2083.codfw.wmnet with reason: host reimage
* 12:09 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2100.codfw.wmnet with reason: host reimage
* 12:00 topranks: drain traffic on cr1-eqiad to allow for line card install and JunOS upgrade [[phab:T426343|T426343]]
* 11:52 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2083.codfw.wmnet with OS trixie
* 11:50 cmooney@dns2005: END - running authdns-update
* 11:49 cmooney@dns2005: START - running authdns-update
* 11:48 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2100.codfw.wmnet with OS trixie
* 11:40 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
* 11:40 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
* 11:36 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply
* 11:31 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db2240: Migration of db2240.codfw.wmnet completed
* 11:30 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/zotero: apply
* 11:28 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/zotero: apply
* 11:27 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:27 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:27 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
* 11:23 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:21 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2240.codfw.wmnet with OS trixie
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 11:20 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:17 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: &quot;Triggered by cookbooks.sre.dns.netbox: add new link IP dns for trasnport circuits to magru - cmooney@cumin1003&quot;
* 11:16 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
* 11:16 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
* 11:15 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2086.codfw.wmnet with OS trixie
* 11:14 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2106.codfw.wmnet with OS trixie
* 11:14 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
* 11:13 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
* 11:12 cmooney@cumin1003: START - Cookbook sre.dns.netbox
* 11:09 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2115.codfw.wmnet with OS trixie
* 11:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 11:00 cwilliams@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2240.codfw.wmnet with reason: host reimage
* 10:53 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:49 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:44 atsuko@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:44 cwilliams@cumin1003: START - Cookbook sre.hosts.reimage for host db2240.codfw.wmnet with OS trixie
* 10:44 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2086.codfw.wmnet with reason: host reimage
* 10:42 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2106.codfw.wmnet with reason: host reimage
* 10:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db2240: Upgrading db2240.codfw.wmnet
* 10:41 cwilliams@cumin1003: dbmaint on s4@codfw [[phab:T429893|T429893]]
* 10:40 cwilliams@cumin1003: START - Cookbook sre.mysql.major-upgrade
* 10:39 atsuko@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2115.codfw.wmnet with reason: host reimage
* 10:27 cwilliams@cumin1003: dbctl commit (dc=all): 'Depool db2240 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94653 and previous config saved to /var/cache/conftool/dbconfig/20260701-102658-cwilliams.json
* 10:26 moritzm: installing nginx security updates
* 10:26 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2086.codfw.wmnet with OS trixie
* 10:23 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2179 to s4 primary [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94652 and previous config saved to /var/cache/conftool/dbconfig/20260701-102356-cwilliams.json
* 10:23 cezmunsta: Starting s4 codfw failover from db2240 to db2179 - [[phab:T430127|T430127]]
* 10:23 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2106.codfw.wmnet with OS trixie
* 10:20 atsuko@cumin2003: START - Cookbook sre.hosts.reimage for host cirrussearch2115.codfw.wmnet with OS trixie
* 10:15 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2179 with weight 0 [[phab:T430127|T430127]]', diff saved to https://phabricator.wikimedia.org/P94651 and previous config saved to /var/cache/conftool/dbconfig/20260701-101531-cwilliams.json
* 10:15 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 40 hosts with reason: Primary switchover s4 [[phab:T430127|T430127]]
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:56 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template (take 2) - oblivian@cumin1003
* 09:55 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template (take 2) - oblivian@cumin1003&quot;
* 09:51 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
* 09:39 mszwarc@deploy1003: Synchronized private/SuggestedInvestigationsSignals/SuggestedInvestigationsSignal4n.php: Update SI signal 4n (duration: 06m 08s)
* 09:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:21 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 09:14 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 09:02 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:54 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:38 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:38 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:36 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.9  refs [[phab:T423918|T423918]]
* 08:21 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1002.wikimedia.org
* 08:21 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] (duration: 36m 11s)
* 08:15 filippo@cumin1003: conftool action : set/pooled=yes; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 08:09 mszwarc@deploy1003: mszwarc, abi: Continuing with deployment
* 08:03 mszwarc@deploy1003: mszwarc, abi: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:55 filippo@cumin1003: conftool action : set/pooled=no; selector: service=dumps-nfs,name=clouddumps1001.wikimedia.org
* 07:51 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
* 07:45 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306850{{!}}ULS rewrite: change description key in EmptySearchEntrypoint (T429882)]]
* 07:30 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050] (duration: 00m 22s)
* 07:29 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train 2nd try [analytics/refinery@410f2050]
* 07:28 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050] (duration: 01m 59s)
* 07:28 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] (duration: 07m 19s)
* 07:26 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (thin): Regular analytics weekly train THIN [analytics/refinery@410f2050]
* 07:26 aqu@deploy1003: Finished deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050] (duration: 04m 32s)
* 07:24 mszwarc@deploy1003: wmde-fisch, mszwarc: Continuing with deployment
* 07:23 mszwarc@deploy1003: wmde-fisch, mszwarc: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:21 aqu@deploy1003: Started deploy [analytics/refinery@410f205]: Regular analytics weekly train [analytics/refinery@410f2050]
* 07:21 aqu@deploy1003: Finished deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050] (duration: 02m 01s)
* 07:20 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306710{{!}}Fix async loading in footnote click interaction experiment (T415904)]], [[gerrit:1306711{{!}}Fix async loading in footnote click interaction experiment (T415904)]]
* 07:19 aqu@deploy1003: Started deploy [analytics/refinery@410f205] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@410f2050]
* 07:13 mszwarc@deploy1003: Finished scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] (duration: 09m 13s)
* 07:09 mszwarc@deploy1003: mszwarc, chlod, revi: Continuing with deployment
* 07:06 mszwarc@deploy1003: mszwarc, chlod, revi: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
* 07:04 mszwarc@deploy1003: Started scap sync-world: Backport for [[gerrit:1306304{{!}}frwiki: change to Wikipedia 25 logo (T430409)]], [[gerrit:1306221{{!}}Temporarily change plwiki tagline for 1.7M articles (T430512)]], [[gerrit:1306649{{!}}CommonSettings: add Ombuds to wgWMCGlobalGroupToRateLimitClass (T430641)]]
* 06:55 elukey: upgrade all trixie hosts to pywmflib 3.0 - [[phab:T430552|T430552]]
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:43 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:43 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:42 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix template - oblivian@cumin1003
* 06:41 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Fix template - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:35 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:34 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:31 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Niharika29 out of all services on: 2453 hosts
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.hiddenparma (exit_code=99) Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:30 oblivian@cumin1003: END (FAIL) - Cookbook sre.deploy.python-code (exit_code=99) hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Various improvements - oblivian@cumin1003
* 06:30 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: &quot;Various improvements - oblivian@cumin1003&quot;
* 06:01 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1039.eqiad.wmnet with reason: issues
* 05:41 marostegui@cumin1003: conftool action : set/weight=100; selector: name=clouddb1027.eqiad.wmnet
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2068.codfw.wmnet with OS trixie
* 05:40 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s2
* 05:40 marostegui@cumin1003: conftool action : set/pooled=yes; selector: name=clouddb1027.eqiad.wmnet,service=s7
* 05:36 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2109.codfw.wmnet with reason: host reimage
* 05:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:16 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2109.codfw.wmnet with OS trixie
* 05:15 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2068.codfw.wmnet with reason: host reimage
* 05:09 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2067.codfw.wmnet with OS trixie
* 04:56 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2068.codfw.wmnet with OS trixie
* 04:49 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:45 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2067.codfw.wmnet with reason: host reimage
* 04:27 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2067.codfw.wmnet with OS trixie
* 03:47 slyngshede@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es1039.eqiad.wmnet with reason: Hardware crash
* 03:21 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:59 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:55 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:51 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2107.codfw.wmnet with reason: host reimage
* 02:35 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:31 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2107.codfw.wmnet with OS trixie
* 02:30 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:26 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2085.codfw.wmnet with reason: host reimage
* 02:22 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch2072.codfw.wmnet with reason: host reimage
* 02:09 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2085.codfw.wmnet with OS trixie
* 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 06m 54s)
* 02:03 ryankemper@cumin2002: START - Cookbook sre.hosts.reimage for host cirrussearch2072.codfw.wmnet with OS trixie
* 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
* 01:07 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad back to read-write - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94649 and previous config saved to /var/cache/conftool/dbconfig/20260701-010716-ladsgroup.json
* 01:05 ladsgroup@dns1004: END - running authdns-update
* 01:05 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depool es1039 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94648 and previous config saved to /var/cache/conftool/dbconfig/20260701-010551-ladsgroup.json
* 01:03 ladsgroup@dns1004: START - running authdns-update
* 01:00 ladsgroup@cumin1003: dbctl commit (dc=all): 'Promote es1035 to es7 primary [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94647 and previous config saved to /var/cache/conftool/dbconfig/20260701-010002-ladsgroup.json
* 00:58 Amir1: Starting es7 eqiad failover from es1039 to es1035 - [[phab:T430765|T430765]]
* 00:53 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es1035 with weight 0 [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94646 and previous config saved to /var/cache/conftool/dbconfig/20260701-005329-ladsgroup.json
* 00:53 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 9 hosts with reason: Primary switchover es7 [[phab:T430765|T430765]]
* 00:42 ladsgroup@cumin1003: dbctl commit (dc=all): 'Set es7 eqiad as read-only for maintenance - [[phab:T430765|T430765]]', diff saved to https://phabricator.wikimedia.org/P94645 and previous config saved to /var/cache/conftool/dbconfig/20260701-004221-ladsgroup.json
* 00:20 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2102.codfw.wmnet with OS trixie
* 00:15 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch2103.codfw.wmnet with OS trixie
* 00:05 dr0ptp4kt: DEPLOYED Refinery at {{Gerrit|4e7a2b32}} for changes: pageview allowlist {{Gerrit|1305158}} (+min.wikiquote) {{Gerrit|1305162}} (+bol.wikipedia), {{Gerrit|1305156}} (+isv.wikipedia); {{Gerrit|1305980}} (pv allowlist -api.wikimedia, sqoop +isvwiki); sqoop {{Gerrit|1295064}} (+globalimagelinks) {{Gerrit|1295069}} (+filerevision) using scap, then deployed onto HDFS (manual copyToLocal required additionally)

== Other archives ==
See [[Server Admin Log/Archives]].

&lt;noinclude&gt;
[[Category:SAL]]
[[Category:Operations]]
&lt;/noinclude&gt;</text>
      <sha1>h1g9s6ypgebuotr3j7e2naizksav1bg</sha1>
    </revision>
  </page>
  <page>
    <title>Data Platform/Data Lake/Edits/MediaWiki history</title>
    <ns>0</ns>
    <id>253925</id>
    <revision>
      <id>2433178</id>
      <parentid>2391314</parentid>
      <timestamp>2026-07-05T00:10:28Z</timestamp>
      <contributor>
        <username>Bamyers99</username>
        <id>2099</id>
      </contributor>
      <comment>/* Changes and known problems */ event_log_id, event_user_is_cross_wiki added</comment>
      <origin>2433178</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="7324" sha1="m5pmhj3p1msxp47atx7j9yr7stav842" xml:space="preserve">This page describes the data set that stores the '''denormalized edit history''' of WMF's wikis. It lives in the [[Analytics/Systems/Cluster|Analytics Hadoop cluster]] and is accessible via the Hive table &lt;code&gt;wmf.mediawiki_history&lt;/code&gt;.

A new snapshot covering all of history is generated from the source data each month. The process is summarized in the diagram below; see [[Analytics/Systems/Data Lake/Edits/Pipeline]] for more details.

&lt;imagemap&gt;
File:Pipeline tree - MediaWiki History and main child datasets.jpg|thumb|center|800px|alt=MediaWiki History and its main child datasets - clickable image|MediaWiki History and main child datasets. '''Tip:''' Most of the boxes are clickable and link to their documentation.

rect 2360 444 3452 1236 [[Data_Platform/Systems/MediaWiki_replicas|Data_Platform/Systems/MediaWiki_replicas]]
rect 4112 456 5244 1248 [[Help:Wiki_Replicas|Wiki Replicas]]
rect 2992 1548 4540 1988 [[Data_Platform/Systems/Edit_data_loading|Data Platform/Systems/Edit data loading]]
rect 956 2708 2332 3488 [[Data_Platform/Data_Lake/Edits]]
rect 3048 2748 4100 3220 [[Data_Platform/Data_Lake/Edits/Geoeditors#Editors_daily|Data Platform/Data Lake/Edits/Geoeditors#Editors_daily]]
rect 4804 2500 5860 2956 [[Data_Platform/Data_Lake/Edits/Geoeditors|Data Platform/Data Lake/Edits/Geoeditors]]
rect 4812 3668 5864 4140 [[Data_Platform/Data_Lake/Edits/Edit_hourly|Data Platform/Data Lake/Edits/Edit hourly]]
rect 4808 4688 5948 5128 [[Data_Platform/Systems/Mediawiki_history_reduced_algorithm|Data Platform/Systems/Mediawiki history reduced algorithm]]
rect 828 3684 2364 4120 [[Data_Platform/Systems/Page_and_user_history_reconstruction|Data Platform/Systems/Page and user history reconstruction]]
rect 6540 2504 7588 2944 [[Data_Platform/Data_Lake/Edits/Geoeditors/Public|Data Platform/Data Lake/Edits/Geoeditors/Public]]
rect 6540 3032 7592 3484 [[Data_Platform/Data_Lake/Edits/MediaWiki_history_dumps|Data Platform/Data Lake/Edits/MediaWiki history dumps]]
rect 6544 4088 7584 4536 [[Data_Platform/Systems/AQS|Data Platform/Systems/AQS]]
rect 6540 4632 7588 5192 [[Data_Platform/Data_Lake/Edits/Mediawiki_history_reduced|Data Platform/Data Lake/Edits/Mediawiki history reduced]]
&lt;/imagemap&gt;

== Public version ==
This data is published as a collection of files on our dumps infrastructure: [[Analytics/Data_Lake/Edits/Mediawiki_history_dumps]].

== Schema ==
For schema documentation, see the [https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,wmf.mediawiki_history,PROD)/Schema entry in DataHub].

== Changes and known problems ==
{| class=&quot;wikitable&quot;
!Date
!Phab
Task
!Snapshot version
!Details
|-
|2026-05-19 
|{{Phabricator|T425986}} 
|2026-05
|event_log_id, event_user_is_cross_wiki added
|-
|2025-10-21
|{{Phabricator|T365648}} 
|2025-11
|user_central_id was added as a stable identifier to uniquely a user account globally across all wikis.
|-
|2023-11-01
|{{Phabricator|T350489}}
|2023-10
|The mediawiki_project_namespace_map table schema was updated.  The update was backwards-compatible but the code used the raw data, superimposing its own schema.  This was the right decision for performance when we created the job, but latest Spark makes this unnecessary.  The job should be updated to use a select statement and future-proof itself.  This has not been prioritized.
|-
|2023-09-01
|{{Phabricator|T344632}}
|2023-08
|A system user, &quot;Global_rename_script&quot;, was given an id and caused a sizeable shift in data.  The checker errors were ignored as false alarms.
|-
|2023-08-03
|{{Phabricator|T345208}}
|2023-07
|Fixes to how redacted actor ids show up on Cloud replicas caused downstream problems in MW history.  Skew-join helper logic was updated and jobs were rerun.  The checker still flagged a sizable difference, probably due to deleted users no longer being seen as valid actors.  It was decided that we should ignore this difference and not vet the data further.
|-
|2022-06-01
|{{Phabricator|T309987}}
|2022-05
|Changes in the production database caused sqoop to break, delays in the mw history job, and delays for all dependent datasets.
|-
|2020-08
|{{Phabricator|T259823}}
|2020-06
|Some page ids are null or zero, and other records appear as duplicates when attempting to use some seemingly unique column combinations
|-
|2019-07
|{{Phabricator|T221825}}
|2019-05
|Schema changes:

* Addition of &lt;code&gt;page_first_edit_timestamp&lt;/code&gt;
* Addition of &lt;code&gt;revision_is_from_before_page_creation&lt;/code&gt;

Improvements in linking more user and page events into full histories, that we were not able to put together before.  Dataset should in general be more consistent and accurate.
|-
|2019-05
|{{Phabricator|T221824}}
|2019-04
|Schema changes:

* Addition of &lt;code&gt;event_user_is_bot_by_historical&lt;/code&gt; and &lt;code&gt;event_user_is_bot_by&lt;/code&gt; as well as &lt;code&gt;user_is_bot_by_historical&lt;/code&gt; and &lt;code&gt;user_is_bot_by&lt;/code&gt;
* Addition of &lt;code&gt;event_user_creation_timestamp&lt;/code&gt;, &lt;code&gt;event_user_first_timestamp&lt;/code&gt; as well as &lt;code&gt;user_creation_timestamp&lt;/code&gt;, &lt;code&gt;user_first_timestamp&lt;/code&gt;. The user registration is the one stored in the user table, the user creation one is retrieved from the logging table (user creation event), and the first-edit is the date of the user first edit, whether deleted or not.
* Removal ('''BREAKING''') &lt;code&gt;of event_user_is_bot_by_name&lt;/code&gt; and &lt;code&gt;user_is_bot_name&lt;/code&gt; (replaced by &lt;code&gt;is_bot_by&lt;/code&gt; above)
* Addition of &lt;code&gt;page_is_deleted&lt;/code&gt;
* Addition of &lt;code&gt;revision_deleted_parts&lt;/code&gt; and &lt;code&gt;revision_deleted_parts_are_suppressed&lt;/code&gt;
* Rename of &lt;code&gt;revision_is_deleted&lt;/code&gt; to &lt;code&gt;revision_is_deleted_by_page_deletion&lt;/code&gt;, and &lt;code&gt;revision_deleted_timestamp&lt;/code&gt; to &lt;code&gt;revision_deleted_by_page_deletion_timestamp.&lt;/code&gt;
* Addition of &lt;code&gt;revision_tags&lt;/code&gt;

Thanks to improvement made on user-history-reconstruction, linking between user-states and page/revision states is now a lot more accurate (see Task T218463).
|-
|2018-10
|{{Phabricator|T209031}}
|2018-10 and 2018-11
|due to the refactor of mediawiki-comments into a separate table, the revision-comments are not available in the table for the two snapshots listed here.
|-
|2017-12
|
|2017-11
|For pairs of fields that give current and historical versions of a value, rename the fields so that &lt;code&gt;_historical&lt;/code&gt; is appended to the historical field rather than &lt;code&gt;_latest&lt;/code&gt; to the current one.
Revisions happening before page-creation date (due to restore over existing page) are now correctly linked.

History of pages with complex delete/restore patterns is on purpose not yet orretly worked. Will happen after Wikistats-2 release.
|-
|2017-06
| {{Phabricator|T161147}}
| 2017-06
|Provide cumulative edit count 
|-
|2017-06
| {{Phabricator|T170493}}
| 2017-06
|Use native timestamps (java.sql.Timestamp, but stillsaves them as JDBC compliant strings)
|-
|2016-10-06
|
|n/a
|The dataset contains data for simplewiki and enwiki until september 2016. Still we need to productionize the automatic updates to that table and import all the wikis.
|-
|2017-03-01
|
|n/a
|Add the &lt;code&gt;snapshot&lt;/code&gt; partition, allowing to keep multiple versions of the history. Data starts to flow regularly (every month) from labs.
|}

[[Category:Edits data]]
[[Category:Data platform]]</text>
      <sha1>m5pmhj3p1msxp47atx7j9yr7stav842</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Wm-bot/SAL</title>
    <ns>498</ns>
    <id>383664</id>
    <revision>
      <id>2433174</id>
      <parentid>2395478</parentid>
      <timestamp>2026-07-04T14:49:48Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>MacFan4000: restarted postgresql</comment>
      <origin>2433174</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="3402" sha1="3y7u50ml184kt9aowz9rslm9nc0zlr2" xml:space="preserve">=== 2026-07-04 ===
* 14:49 MacFan4000: restarted postgresql

=== 2026-03-24 ===
* 22:18 legoktm: added addshore, per request

=== 2025-09-23 ===
* 17:22 MacFan4000: cleared excess apache logs and set logrotate to be a bit more aggressive

=== 2025-01-14 ===
* 16:58 MacFan4000: reboot wm-bot-bookworm

=== 2024-07-18 ===
* 17:21 MacFan4000: deleted unused buster instances

=== 2024-06-20 ===
* 16:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)
* 16:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs
* 16:30 dhinus: restarted after crash

=== 2024-03-29 ===
* 17:58 MacFan4000: full restart following disconnection

=== 2023-12-21 ===
* 18:10 MacFan4000: full restart of bot + xmlrcs following disconnection

=== 2023-06-29 ===
* 17:40 MacFan4000: completed bookworm migration

=== 2023-06-26 ===
* 17:22 MacFan4000: migrating pgsql data over to wm-bot-pg2-bookworm
* 01:51 MacFan4000: created two bookworm instances

=== 2023-06-24 ===
* 16:07 MacFan4000: performing updates on wm-bot-pg2

=== 2023-02-13 ===
* 19:56 MacFan4000: full restart following outage

=== 2023-01-17 ===
* 18:27 MacFan4000: full restart due to disconnection

=== 2022-12-13 ===
* 03:16 MacFan4000: locally applied fixes to stop the apache error log from filling up with spam

=== 2022-12-12 ===
* 23:49 MacFan4000: clear apache logs to free up space

=== 2022-11-21 ===
* 20:09 MacFan4000: updated xmlrcs url in config

=== 2022-10-19 ===
* 23:02 legoktm: Added `AddDefaultCharset utf-8` to apache config on bawolff's request

=== 2022-08-16 ===
* 20:31 MacFan4000: hard restart (killed procs + removed pids, then ran restart) due to bot failing to reconnect

=== 2022-08-03 ===
* 21:04 MacFan4000: manually fix nicks via telnet

=== 2022-04-19 ===
* 18:06 MacFan4000: updated a cloak in the admins config

=== 2022-04-05 ===
* 14:28 MacFan4000: fix nick of an instance via telnet

=== 2022-02-15 ===
* 03:28 MacFan4000: updated cloak in admins config

=== 2021-06-15 ===
* 15:28 MacFan4000: copied freenode channel config for #wikimedia-fundraising to libera
* 12:54 MacFan4000: killed a few lingering connections to freenode (wm-bot on freenode is now discontinued)

=== 2021-06-14 ===
* 21:29 MacFan4000: copied freenode config for #cvn-sw to libera

=== 2021-06-12 ===
* 23:56 MacFan4000: copied channel config for #cvn-es-scan, #cvn-wp-fa and #countervandalism

=== 2021-05-28 ===
* 20:08 legoktm: added billinghurst as project admin

=== 2021-01-28 ===
* 00:01 MacFan4000: (logging) Restarted from #MacFan4000 where there was a functioning instance

=== 2020-10-06 ===
* 06:01 Matthew: Restarted bot completely, two instances refused to reconnect.

=== 2020-05-19 ===
* 15:10 MacFan4000: did @system-rejoin-all and @restart from #MacFan4000, as the bot wasn't responding in #wm-bot

=== 2020-03-08 ===
* 04:30 bd808: Deleted jessie instances wm-bot-pg and wm-bot2 ([[phab:T236564|T236564]])

=== 2020-01-07 ===
* 15:09 jeh: log archive missing for some channels since 2019-12-16, restarted wm-bot service on wm-bot2

=== 2019-06-24 ===
* 11:02 arturo: updated some SQL data

=== 2018-12-11 ===
* 15:06 andrewbogott: moving project to eqiad1-r

=== 2018-06-28 ===
* 21:00 bd808: Restarted apache2 on wm-bot2.wm-bot.eqiad.wmflabs; web interface to logs was unresponsive

{{SAL|Project Name=wm-bot}}
&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>3y7u50ml184kt9aowz9rslm9nc0zlr2</sha1>
    </revision>
  </page>
  <page>
    <title>Nova Resource:Tools.lexeme-forms/SAL</title>
    <ns>498</ns>
    <id>443946</id>
    <revision>
      <id>2433173</id>
      <parentid>2433172</parentid>
      <timestamp>2026-07-04T13:54:30Z</timestamp>
      <contributor>
        <username>Stashbot</username>
        <id>7414</id>
      </contributor>
      <comment>wmbot~lucaswerkmeister@tools-bastion-15: deployed b304cd29af (temporary T431146 branch: log request ID and details of every refresh token response)</comment>
      <origin>2433173</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="81062" sha1="g7m9sh7r0a91dfo7mi59fvhjd3zpu5t" xml:space="preserve">=== 2026-07-04 ===
* 13:54 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|b304cd29af}} (temporary [[phab:T431146|T431146]] branch: log request ID and details of every refresh token response)
* 11:07 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|593bb47c56}} (temporary [[phab:T431146|T431146]] branch: enable debug logging for requests_oauthlib.oauth2_session)

=== 2026-06-30 ===
* 19:01 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|bab5e77be1}} (mwoauth2 0.1.0)

=== 2026-06-18 ===
* 20:20 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|279d1068c2}} (l10n updates: ar)

=== 2026-06-13 ===
* 16:16 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|d153685969}} (add login check)
* 16:13 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5fde4e8ec2}} (use authenticated session for Wikifunctions calls: [[phab:T349966|T349966]], [[phab:T423542|T423542]])

=== 2026-06-11 ===
* 19:59 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3358ae7e8c}} (l10n updates: ar, fi, frp, it, ko, nl, sk, zh-hans; using the previously deployed {{GENDER:}} support in duplicates-instructions)

=== 2026-06-06 ===
* 19:36 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|22a11f62c4}} (add {{GENDER:}} support to duplicates-instructions message)

=== 2026-06-04 ===
* 19:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3c09a6963f}} (l10n updates: nl)

=== 2026-06-03 ===
* 17:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fd3d67655e}} (bump mwoauth2)

=== 2026-05-29 ===
* 17:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|11f47f0f24}} (install mwoauth2 as package)

=== 2026-05-27 ===
* 21:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|994cfa51b3}} (make mwoauth2 strictly typed)
* 18:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|73fcbaa464}} (extract mwoauth2 module)

=== 2026-05-25 ===
* 13:51 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c163567ccd}} (l10n updates: nb)

=== 2026-05-21 ===
* 21:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|97d0866c92}} (treat outdated [OAuth 1] access tokens more robustly)
* 19:18 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f0d7740038}} (l10n updates: ko, ta, vi)

=== 2026-05-20 ===
* 18:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|dfc101f3e0}} (Python 3.14, aka 𝜋thon)
* 18:50 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5b92917005}} (upgrade dependencies)
* 18:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|a42e65b596}} (configure Gunicorn --forwarded-allow-ips)

=== 2026-05-19 ===
* 17:59 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|43b18b8299}} (prevent OAuthLib InsecureTransportError more strongly)
* 17:52 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|80583945a8}} (migrate to OAuth 2)

=== 2026-04-25 ===
* 23:34 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|4f201ca05a}} (update gunicorn config)
* 19:30 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ab0ee1c0ce}} (Russian imperfective verbs)

=== 2026-04-13 ===
* 17:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fe623b17c0}} (l10n updates: hi, ms-arab)

=== 2026-04-06 ===
* 12:36 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|68d0059881}} (l10n updates: ms-arab)

=== 2026-03-26 ===
* 17:21 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|a2dde3c334}} (l10n updates: pa)

=== 2026-03-23 ===
* 19:02 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|03da8bff9b}} (l10n updates: kea)

=== 2026-03-19 ===
* 13:17 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|cdc85da9aa}} (l10n updates: kea)

=== 2026-03-16 ===
* 13:26 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|83b82c6ea4}} (l10n updates: ary, ga)

=== 2026-03-14 ===
* 14:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2e36547a31}} (Moroccan Arabic templates – part of Wikimedia Hackathon Northwestern Europe 2026 \o/)

=== 2026-03-12 ===
* 16:31 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|0282ad0864}} (l10n updates: kea)

=== 2026-03-09 ===
* 13:22 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f0983e0fcf}} (l10n updates: kea)

=== 2026-02-10 ===
* 21:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ca57a0da6a}} (noop – update a test)

=== 2026-02-09 ===
* 18:40 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|49f7ed4319}} (l10n updates: el)

=== 2026-02-02 ===
* 13:44 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|74bb77b1b4}} (l10n updates: el, pl)

=== 2026-01-22 ===
* 12:56 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|4bdcac2b61}} (l10n updates: pa)

=== 2026-01-18 ===
* 17:45 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|3ba43046ee}} (avoid changing lemma if not necessary)

=== 2026-01-05 ===
* 12:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|fa52e74def}} (l10n updates: id)

=== 2026-01-03 ===
* 17:03 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|2825501536}} (l10n updates: ca, it, pl, sv)

=== 2025-12-22 ===
* 19:08 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|40105a8e8b}} (l10n updates: it, vi)

=== 2025-12-11 ===
* 21:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|5c7dd452be}} (l10n updates: mk)

=== 2025-12-01 ===
* 13:01 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|54c6749c45}} (l10n updates: fi)

=== 2025-11-19 ===
* 18:55 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|7f658a6675}} (update Bootstrap)

=== 2025-11-18 ===
* 20:04 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|c74254856c}} (fix skiplink visibility)

=== 2025-11-17 ===
* 13:06 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|34ee7bf7ac}} (l10n updates: cy)

=== 2025-11-13 ===
* 12:46 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|808bd196dc}} (l10n updates: anp)

=== 2025-11-10 ===
* 19:04 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|730ae77335}} (drop typing_extensions)
* 18:58 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|36b8ab588c}} (upgrade dependencies)
* 18:47 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|422cb1f05c}} (l10n updates: frp, ro)

=== 2025-10-27 ===
* 14:15 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|40579b07e7}} (l10n updates: el, tg)

=== 2025-10-20 ===
* 12:30 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|9d7035ced8}} (l10n updates: frp)

=== 2025-10-13 ===
* 18:10 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|ffc43e9936}} (l10n updates: lb)

=== 2025-09-29 ===
* 18:23 wmbot~lucaswerkmeister@tools-bastion-15: deployed {{Gerrit|f458d2938d}} (l10n updates: ko-kp)

=== 2025-09-11 ===
* 18:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|afc726918e}} (l10n updates: rki)

=== 2025-09-04 ===
* 15:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c35d575859}} (l10n updates: nb)

=== 2025-09-01 ===
* 18:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1e9e985c5d}} (l10n updates: vi)

=== 2025-08-25 ===
* 17:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|457dd44066}} (l10n updates: aig, pt)

=== 2025-08-24 ===
* 22:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3669a5db51}} (upgrade dependencies, including PyMySQL 1.1.2 with Python 3.13 compatibility)

=== 2025-08-21 ===
* 19:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3776ee4000}} (l10n updates: yue-hant)

=== 2025-08-18 ===
* 19:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9b5fc1cef3}} (Portuguese Wikifunctions)
* 19:07 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2886086be9}} (add missing wikifunctions_intro to german-noun-masculine)
* 17:57 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|243f584c59}} (l10n updates: ar, tg, yue-hant)

=== 2025-08-14 ===
* 16:25 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d4ef0aa38c}} (l10n updates: yue-hant)

=== 2025-08-07 ===
* 19:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ebeea040cb}} (l10n updates: yue-hant)

=== 2025-07-31 ===
* 17:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6b1e8c15ae}} (l10n updates: pt, pt-br, sl)

=== 2025-07-24 ===
* 19:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ece1469a65}} (l10n updates: pt)

=== 2025-07-17 ===
* 19:29 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|672ec5bee8}} (l10n updates: yue-hant, zh-hant)

=== 2025-07-13 ===
* 15:50 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3c977ccc7b}} (specify .python-version)
* 14:46 lucaswerkmeister: disregard the previous message, wrong tool 🤦
* 14:46 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;OAUTH&quot;][&quot;CONSUMER_KEY&quot;]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY

=== 2025-07-12 ===
* 21:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|336dd318ca}} (upgrade to Python 3.13)
* 20:56 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|dd03e95876}} (update documentation; no-op deployment, just to test the new buildservice procedure and put it in a single command)
* 20:50 wmbot~lucaswerkmeister@tools-bastion-13: cp www-unused-tool-now-runs-on-buildservice/python/src/service.template .
* 20:49 wmbot~lucaswerkmeister@tools-bastion-13: mv www www-unused-tool-now-runs-on-buildservice
* 20:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6d259028e}} (successful migration to buildservice)
* 18:09 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|0bf37ac8d8}} (tried but failed to migrate to build service [OSError: No username set in the environment], will try again later, for now running in python3.11 again)
* 17:42 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f8033bda4a}} (read config from envvars)
* 17:41 lucaswerkmeister: commented out config.yaml, should use envvars instead
* 17:41 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;SECRET_KEY&quot;]))' {{!}} toolforge envvars create TOOL_SECRET_KEY
* 17:40 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;OAUTH&quot;][&quot;CONSUMER_SECRET&quot;]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_SECRET
* 17:40 lucaswerkmeister: python3 -c 'import yaml; print(yaml.safe_dump(yaml.safe_load(open(&quot;config.yaml&quot;))[&quot;OAUTH&quot;][&quot;CONSUMER_KEY&quot;]))' {{!}} toolforge envvars create TOOL_OAUTH__CONSUMER_KEY
* 17:16 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|15261afefb}} (change config keys to uppercase to work around [[phab:T374780|T374780]])

=== 2025-07-10 ===
* 19:59 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8c32f8b90c}} (l10n updates: hu)

=== 2025-07-07 ===
* 06:33 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cc28ff0494}} (l10n updates: et, it, nn, pt-br, ru)

=== 2025-06-16 ===
* 17:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c7b450ab94}} (update code for newer mwapi version)

=== 2025-06-11 ===
* 12:23 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cae8c3c341}} (upgrade dependencies, including toolforge 6.1.0; use toolforge.load_private_yaml() from [[phab:T333728|T333728]])

=== 2025-05-31 ===
* 13:53 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|706110e863}} (l10n updates: da, lb)

=== 2025-05-13 ===
* 16:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8a583bf6ff}} (l10n updates: tg)

=== 2025-05-06 ===
* 17:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|7349295f62}} (l10n updates: el)
* 17:24 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e969f66351}} (update absolute_construction item ID)

=== 2025-04-22 ===
* 23:10 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d9516b6b1c}} (Quechua verb Wikifunctions)

=== 2025-04-21 ===
* 18:20 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|c6d552c9c3}} (upgrade dependencies, including toolforge-i18n 0.1.2)

=== 2025-04-19 ===
* 10:56 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5425b40c0f}} (l10n updates: es)

=== 2025-04-14 ===
* 19:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ae10863f8f}} (l10n updates: af)

=== 2025-04-07 ===
* 18:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e106b7b684}} (Quechua verbs + l10n updates: es, pa, qu, zh-hant)

=== 2025-04-04 ===
* 19:48 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a377a0be8c}} (remove unneeded CSS)

=== 2025-03-29 ===
* 21:16 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|98e408e5a6}} (Russian perfective verbs)

=== 2025-03-15 ===
* 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|ab6621b22d}} (l10n updates: ar)

=== 2025-03-11 ===
* 20:35 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6def84813}} (l10n updates: lb)

=== 2025-02-21 ===
* 20:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|81611bc5dc}} (l10n updates: pa, tr)

=== 2025-02-04 ===
* 21:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|2ccb28ad17}} (l10n updates: lb)

=== 2025-01-24 ===
* 10:21 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|223cafa209}} (l10n updates: ms)

=== 2025-01-09 ===
* 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|cebad0e4dd}} (l10n updates: ia, pa)

=== 2025-01-06 ===
* 20:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e7e3f2a500}} (l10n updates: cs, he)

=== 2024-12-21 ===
* 22:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|eb9d0ae3c2}} (l10n updates: lb, pa; also upgrade dependencies, including Flask 3.1.0 and Jinja2 3.1.5)

=== 2024-12-12 ===
* 22:13 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5ffdfb2c55}} (l10n updates: he, nl)

=== 2024-11-18 ===
* 19:47 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|3933dbfa7f}} (l10n updates: af, ar, de, fr, gl, he, krc, mk, pa, sk, zh-hans); manually restored sh-latn ([[phab:T379188|T379188]])

=== 2024-11-04 ===
* 17:32 wmbot~lucaswerkmeister@tools-bastion-13: webservice stop; webservice start # [[phab:T378976|T378976]]

=== 2024-11-02 ===
* 16:54 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a6768b885c}} (add setting for using Wikifunctions)
* 14:01 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4fdd9491ee}} (improve Wikifunctions UI)
* 09:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f28c3414ad}} (upgrade dependencies, including Werkzeug 3.1.0); also upgraded pip from 24.2 to 24.3.1

=== 2024-10-25 ===
* 19:30 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8cdbda6ce3}} (upgrade dependencies, including Werkzeug 3.0.6)

=== 2024-10-13 ===
* 11:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bfcaca2fa3}} (upgrade dependencies, including MarkupSafe 3.0)

=== 2024-10-03 ===
* 16:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e6377a9095}} (upgrade dependencies, including toolforge_i18n 0.1.1 and Werkzeug 3.0.4)
* 13:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|a81b469204}} (l10n updates: ms-arab)

=== 2024-09-26 ===
* 15:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|96f45731db}} (l10n updates: ar, ms-arab)

=== 2024-09-11 ===
* 21:00 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|38b3b281ed}} (fix two ZIDs for Breton templates)

=== 2024-09-01 ===
* 14:32 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|53a1efcc14}} (l10n updates: cy, uk)

=== 2024-08-18 ===
* 12:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|309b33b80b}} (l10n updates: pl, tg)
* 12:02 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|6deace1e36}} (Italian masculine+feminine nouns, dependency upgrades)

=== 2024-08-12 ===
* 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|246f9d26da}} (l10n updates: tg)

=== 2024-08-05 ===
* 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e3448958a0}} (upgrade toolforge_i18n to 0.0.7)

=== 2024-07-31 ===
* 19:15 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4775170045}} (upgrade toolforge_i18n to 0.0.6; also upgrade pip to 24.2)

=== 2024-07-26 ===
* 21:11 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|bb61fc3c89}} (l10n updates: vi [no actual translation changes, one addition to the authors, presumably their edit got reverted])

=== 2024-07-22 ===
* 18:27 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|13c4824e3a}} (change Babel code of kaa from kk to uz)

=== 2024-07-21 ===
* 18:12 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e856c9b2d2}} (upgrade toolforge_i18n to 0.0.5; also upgrade pip to 24.1.2)

=== 2024-07-08 ===
* 18:03 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|d6fa2d82b8}} (l10n updates: ja)

=== 2024-07-07 ===
* 18:22 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f3b3981ec9}} (upgrade toolforge_i18n to 0.0.2; also upgrade pip from 24.0 to 24.1.1)

=== 2024-07-05 ===
* 12:28 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1013a7234d}} (l10n updates: ar, de, uk)

=== 2024-06-18 ===
* 19:05 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|8530f5f235}} (l10n updates: eo, fa, kaa, lb)

=== 2024-06-15 ===
* 13:58 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|9cb9b3dfde}} (install toolforge_i18n from PyPI)

=== 2024-06-07 ===
* 09:06 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|253d1b0f45}} (l10n updates: pa)

=== 2024-05-26 ===
* 13:49 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|48a5585566}} (support opting out of Wikifunctions mode)

=== 2024-05-20 ===
* 13:34 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|4d952df88b}} (l10n updates: ms)

=== 2024-05-13 ===
* 18:19 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1c3d80a5e6}} (l10n updates: eu, zh-hans)

=== 2024-05-11 ===
* 12:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bfccf1614c}} (more Hebrew verb templates)

=== 2024-05-09 ===
* 15:40 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|5b88dd1ce1}} (improve toolforge_i18n and upgrade dependencies for newer Babel and Werkzeug)

=== 2024-05-06 ===
* 17:04 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c5618f5968}} (set bot flag in bulk mode)
* 15:43 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8fa2740a72}} (README update, pulled without webservice restart)

=== 2024-05-05 ===
* 11:47 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|400cc9cb84}} (update Hebrew pa'al verbs)
* 11:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|19c8210d68}} (Hebrew pa'al verbs)

=== 2024-05-04 ===
* 12:17 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|deb5b1c44e}} (extract toolforge_i18n library: [[phab:T363626|T363626]])

=== 2024-05-03 ===
* 17:08 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|89c98da81f}} (upgrade dependencies for Python 3.12 compat; also upgraded pip&lt;nowiki&gt;{&lt;/nowiki&gt;,-tools&lt;nowiki&gt;}&lt;/nowiki&gt; and wheel while I’m at it)

=== 2024-04-22 ===
* 20:38 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|1be060cd5c}} (l10n updates: ja)

=== 2024-04-18 ===
* 19:52 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|f1a2cd1995}} (use public WikiLambda API)

=== 2024-04-17 ===
* 19:44 wmbot~lucaswerkmeister@tools-bastion-13: deployed {{Gerrit|e5d2281cea}} (l10n updates: krc)
* 18:13 wmbot~lucaswerkmeister@tools-bastion-13: pulled {{Gerrit|fa6c094165}} (templates CC BY-SA 3.0 → 4.0; no webservice restart needed)

=== 2024-04-08 ===
* 17:58 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|559eb5bc47}} (make session permanent after login)

=== 2024-04-06 ===
* 13:35 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|1569542ce6}} (l10n updates: el, fa, zh-hant)

=== 2024-03-24 ===
* 12:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|b630198d56}} (l10n updates: fi, ms-arab)

=== 2024-03-15 ===
* 19:41 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|272a303c09}} (Danish adverbs)
* 16:33 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|8f4985e682}} (improve tests; should have no production impact but I pulled+restarted anyway ^^)

=== 2024-03-10 ===
* 18:40 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|c62a9c1927}} (Maltese templates, including support for non-first forms to be the lemma: Maltese nouns have the third person singular as the lemma)
* 12:42 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|bf88439696}} (l10n updates: fi, ko)

=== 2024-03-04 ===
* 18:12 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|e7a659802c}} (l10n updates: ar, io, lb)

=== 2024-03-03 ===
* 00:26 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|4106259494}} (l10n updates: ht, hu)

=== 2024-02-28 ===
* 18:50 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|3030faaa3c}} (health-check-path, [[phab:T341919|T341919]])

=== 2024-02-23 ===
* 20:21 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|968078dbcd}} (l10n updates: hu, lt) [relog from 19:35 UTC, stashbot had problems]

=== 2024-02-17 ===
* 10:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|f88f2445fc}} (Esperanto adjective+verb Wikifunctions)

=== 2024-02-13 ===
* 18:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|85b6ec6534}} (l10n updates: ja, kaa)

=== 2024-02-07 ===
* 17:59 wmbot~lucaswerkmeister@tools-sgebastion-10: started webservice again (and patched the startup probe into it); took a while to come up but now it seems to be working
* 17:49 wmbot~lucaswerkmeister@tools-sgebastion-10: stopped webservice, restart wasn’t working so let’s try harder
* 17:45 wmbot~lucaswerkmeister@tools-sgebastion-10: restarted webservice, log was full of various errors

=== 2024-02-06 ===
* 20:39 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|344fd43224}} (update Breton noun Wikifunctions)

=== 2024-01-31 ===
* 19:13 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|604b43e316}} (l10n updates: it)

=== 2024-01-26 ===
* 19:03 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|249d9da0b7}} (l10n updates: id, kaa, ru, th)
* 00:22 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|886d99636e}} (more Esperanto noun Wikifunctions)

=== 2024-01-22 ===
* 18:34 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|0b062cafa9}} (Norwegian language name templates)

=== 2024-01-13 ===
* 15:51 wmbot~lucaswerkmeister@tools-sgebastion-10: deployed {{Gerrit|d24dc99256}} (l10n updates: ar)

=== 2024-01-07 ===
* 13:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a97ab796ea}} (wikifunctions: first form from lemma, if missing)

=== 2024-01-06 ===
* 16:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea6b02ac57}} (Wikifunctions returning lists, Z11991→Z12689)

=== 2024-01-04 ===
* 12:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|82f5578b9a}} (l10n updates: ca, de, pl)

=== 2023-12-30 ===
* 15:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5baa3871d0}} (l10n updates: lb, zh-hans)

=== 2023-12-28 ===
* 10:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|45b698823a}} (update Italian adjectives)
* 10:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e68d80748}} (i18n updates: uk)

=== 2023-12-17 ===
* 18:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7611d4e980}} (l10n updates: ia, krc, sv)

=== 2023-12-11 ===
* 18:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|424615e192}} (l10n updates: de, krc, lb, nl, pnb)

=== 2023-12-09 ===
* 16:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fdc7c853c4}} (update Breton noun Wikifunctions)

=== 2023-12-05 ===
* 19:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|95ee032c68}} (l10n updates: ca, hno, io, it, pnb, sl, tr; i18n test improvements and fixes)

=== 2023-12-01 ===
* 19:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ba19a1cd5f}} (l10n updates: ja, sk, zh-hans)
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7acef657d0}} (update Croation noun Wikifunctions)

=== 2023-11-29 ===
* 17:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|54a614fd41}} (fix some spacing)

=== 2023-11-25 ===
* 12:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|171fc2ea54}} (l10n updates: br)

=== 2023-11-19 ===
* 16:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0416376e58}} (German masculine noun Wikifunctions)
* 15:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|11e7d12745}} (one more set of German neuter noun Wikifunctions)
* 13:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|442f510a5b}} (German neuter noun Wikifunctions)

=== 2023-11-18 ===
* 17:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8c123e032e}} (l10n updates: br, he, ko)

=== 2023-11-12 ===
* 17:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cc2cf0ceaf}} (l10n updates: bn, fa, fr, gl, it, lb, mk, nb, vi, zh-hans, zh-hant; yue removed, existing settings are automatically replaced with zh-hant)

=== 2023-11-04 ===
* 18:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|203bc87b5b}} (more German feminine noun Wikifunctions – m/n will follow later)
* 12:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfa1ad40e0}} (first German Wikifunctions: feminine noun -(e)n plural)
* 10:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|365c7e2814}} (cache Wikifunctions results)

=== 2023-11-01 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|240a228f49}} (tests for Wikifunctions, pulled without webservice restart)
* 18:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|92a91137e6}} (Wikifunctions for Breton nouns)

=== 2023-10-30 ===
* 19:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bea713bc0c}} (l10n updates: br)
* 00:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f33e56597c}} (update French Wikifunctions button label)

=== 2023-10-29 ===
* 17:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cca1b1af23}} (Wikifunctions support in edit mode)
* 16:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b5af35ab2b}} (fix Croatian feminine noun instrumental plural)
* 16:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|052ba84de7}} (fix crash for users without Wikifunctions account)
* 15:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c3dc0dd6d}} (experimental Wikifunctions for Esperanto nouns, nominative plural only)
* 14:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0ab3c10890}} (fix Wikifunctions buttons lang= and dir=)
* 14:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5657d03fbb}} (experimental Wikifunctions for French nouns)
* 14:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a64b857485}} (experimental Wikifunctions for Croatian nouns)
* 14:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|40b0df49ee}} (experimental Wikifunctions support – happy birthday Wikidata 🎉)

=== 2023-10-28 ===
* 22:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c1f7a335e8}} (fix input patterns)

=== 2023-10-25 ===
* 17:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cdb1d34e11}} (Werkzeug 3.0.1)

=== 2023-10-20 ===
* 17:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|df7cf04757}} (i18n updates: io, ms-arab)

=== 2023-10-10 ===
* 19:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ad16425ee2}} (l10n updates: nl, uk, zh-hans)

=== 2023-10-06 ===
* 17:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72e12c5a2c}} (l10n updates: zh-hans) + remove hardcoded support for Karai-karai now that MediaWiki has it

=== 2023-10-01 ===
* 17:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|216afb45fa}} (update dependencies, Flask+Werkzeug 3)

=== 2023-09-24 ===
* 13:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5ae3295bb}} (Babel language code of Aragonese, to silence log warnings)
* 13:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|45aa8fe43b}} (Danish proper nouns)

=== 2023-09-22 ===
* 16:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72c20b3b3e}} (l10n updates: cs, kai [new, with temporary hacks], tr, zh ⇒ zh-hans)

=== 2023-09-04 ===
* 16:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|85d978855f}} (Italian adverbs)

=== 2023-08-28 ===
* 18:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|48e3991eb6}} (fix typo in armenian-noun-singulare-tantum)

=== 2023-08-27 ===
* 14:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea49f8c2c7}} (update dependencies)
* 13:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|05522cee84}} (update Italian)

=== 2023-08-24 ===
* 17:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c19c9624ba}} (l10n updates: ca, fa, io)

=== 2023-08-12 ===
* 11:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e0cf031e70}} (l10n updates: it)

=== 2023-08-08 ===
* 18:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|56acd0944a}} (l10n updates: tr)

=== 2023-07-27 ===
* 12:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5d374c3787}} (l10n updates: ban, de, gl)

=== 2023-07-19 ===
* 12:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4fa53fae89}} (l10n updates: pt-br)

=== 2023-07-18 ===
* 08:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|474e48d752}} (update Breton grammatical feature)

=== 2023-07-15 ===
* 12:03 wm-bot: &lt;lucaswerkmeister&gt; pip-sync (i.e., actually install dependencies in the new venv, which I completely forgot to do earlier)
* 11:31 wm-bot: &lt;lucaswerkmeister&gt; kubectl patch deployment lexeme-forms --patch-file patch-add-startup-probe.yml
* 11:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|02f72f81a2}} (Python 3.11)

=== 2023-07-13 ===
* 13:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d7fea069ba}} (l10n updates: pl)

=== 2023-07-10 ===
* 17:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|42679bb5dc}} (l10n updates: yue)

=== 2023-07-09 ===
* 14:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|78711ad373}} (l10n updates: ms)

=== 2023-07-02 ===
* 13:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e2653cf19}} (revert recent punjabi-noun-masculine-guru change)
* 12:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c9e84dfb8d}} (add separators to Dutch nouns)

=== 2023-06-30 ===
* 18:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1ed453b5d5}} (l10n updates: sh → sh-latn, tt → tt-cyrl, [[phab:T336606|T336606]])

=== 2023-06-27 ===
* 20:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fe5983c571}} (l10n updates: ba)

=== 2023-06-25 ===
* 14:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3ad131b7bf}} (Aragonese common nouns)

=== 2023-06-24 ===
* 09:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|213bfabfb4}} (underline links on hover again)

=== 2023-06-22 ===
* 20:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|63c042d9b3}} (l10n updates: it)

=== 2023-06-20 ===
* 18:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7081d2769e}} (support language fallback and ?uselang)
* 17:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e76345eb5}} (l10n updates: ba, id, nb, xmf)

=== 2023-06-18 ===
* 11:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bebc116e22}} (Bootstrap 5)
* 11:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d53e455ef7}} (update Malayalam nouns and add adjective template)

=== 2023-06-16 ===
* 17:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|248590aeb0}} (l10n updates: ba, id, pl)

=== 2023-06-13 ===
* 17:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e9112d022e}} (l10n updates: es)

=== 2023-06-11 ===
* 11:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fb8c4a30ff}} (update punjabi-noun-masculine-guru)

=== 2023-06-09 ===
* 16:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e059c8bbd6}} (l10n updates: fi); also, last time I forgot to git rebase, so this actually includes {{Gerrit|2035050d28}} (l10n updates: sv) as well

=== 2023-06-07 ===
* 07:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2035050d28}} (l10n updates: sv)

=== 2023-06-04 ===
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|08962e4902}} (update past transgressive item ID after merge; only affects czech-verb-perfective)

=== 2023-05-31 ===
* 20:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1ec8c72304}} (Russian adjectivse: remove compound lexical categories)

=== 2023-05-29 ===
* 15:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a5e90a0e02}} (update dependencies)

=== 2023-05-27 ===
* 19:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|07deb7a083}} (Punjabi additive double causative verbs)
* 17:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|889b4ce276}} (Punjabi additive causative verbs)
* 15:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|467d5b9f34}} (Punjabi transitive verbs)
* 15:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6c76e2d3b5}} (fix two Punjabi placeholders)

=== 2023-05-25 ===
* 20:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a50e668166}} (l10n updates: ca, es, fa, fi, ru, tr, ur)

=== 2023-05-19 ===
* 17:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b59c2f0aad}} (l10n updates: es, hi, zh-hant)
* 16:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b80c8ff9db}} (fix “logged in” indicator in several languages)

=== 2023-05-18 ===
* 08:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7f76b0f203}} (l10n updates: br, de, fr, he, hi, hno, ia, mk, pa, pnb, ru, sa, sl, ur)

=== 2023-05-13 ===
* 17:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7d3ab49b06}} (l10n updates: ar, bn, de, eo, fa, fi, fr, he, hy, ia, it, ja, ko, mk, ms, nb, pnb, ru, skr-arab, sl, zh-hant)
* 12:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dfcf34ed51}} (make “logged in as” translatable)
* 11:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|65cb94f3c7}} (punjabi-verb-basic-intransitive templates)

=== 2023-05-12 ===
* 20:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|15b7403971}} (fix stray character)

=== 2023-05-08 ===
* 21:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|db3dd67b8a}} (make more translations available and tweak Babel language codes)
* 20:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7bf757be9}} (fix message keys broken by previous deployment)
* 20:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5f01c59794}} (refactor message keys from _ to -, should make no difference)
* 19:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b51f930220}} (user interface language setting)

=== 2023-05-05 ===
* 12:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72a006c6ea}} (l10n updates: mrh, ta)

=== 2023-05-02 ===
* 00:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5f83647d21}} (test-only change, pulled without webservice restart)

=== 2023-05-01 ===
* 23:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|88da33ddc5}} (GitHub actions only change, pulled without webservice restart)
* 17:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1380884cce}} (upgrade dependescies, GHSA-m2qf-hxjv-5gpq)
* 15:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|75230357a4}} (l10n updates: lt)
* 15:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1554678038}} (improve matching.py for upcoming templates, should make no difference at the moment)
* 14:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|692e255a50}} (refactor matching.py, should make no difference)

=== 2023-04-30 ===
* 15:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|db66a9373c}} (refactor statement groups; should make no difference)

=== 2023-04-25 ===
* 21:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9059e45cda}} (update dependencies, Werkzeug 2.3.0 / Flask 2.3.1)
* 18:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c6dc908e1e}} (refactoring for somevalue support, should make no difference yet)

=== 2023-04-24 ===
* 19:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d5b5c8994f}} (preparation &amp; refactoring, no visible changes)

=== 2023-04-23 ===
* 18:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0f96d60736}} (Punjabi adverbs)
* 18:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|75af96b851}} (Punjabi adjectives)
* 15:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|934f5cffdb}} (Yoruba adjectives)

=== 2023-04-22 ===
* 16:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a074fd9c64}} (trim spaces)
* 15:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fdb0552957}} (remove spaces)
* 15:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b6a1268b21}} (Punjabi nouns)

=== 2023-04-15 ===
* 15:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|604df5c72e}} (two more variables)
* 15:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1b999f4661}} (use variables for entity IDs; should make no difference at runtime)
* 14:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|24fb20fd19}} (sort sets for JSON output)

=== 2023-04-12 ===
* 20:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5b07592a7e}} (two style improvements)

=== 2023-04-10 ===
* 17:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|282a7b6b18}} (l10n updates: anp; currently skipped because unsupported by Babel)

=== 2023-04-08 ===
* 11:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|994cbd48b0}} (fix typo in a Hindustani template)

=== 2023-04-01 ===
* 18:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|08ac04d468}} (fix Hindko template order)

=== 2023-03-22 ===
* 20:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b40cefa378}} (change Hindko templates to hno)

=== 2023-03-19 ===
* 21:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cf1e031a43}} (l10n updates: fi, tt)

=== 2023-03-13 ===
* 21:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d7ba3ddc23}} (l10n updates: hi, pa, tt, ur)

=== 2023-03-08 ===
* 22:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8da3525baf}} (fix lowercase item ID in portuguese-noun-biform)
* 22:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|de17c6bdf6}} (fix hindustani-verb-additive-causative-double-ur label)
* 22:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a99078e1c5}} (hindustani-verb-additive-causative-double templates)

=== 2023-03-06 ===
* 21:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8828e3269e}} (l10n updates: tt)
* 21:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2cbf107d6e}} (hindustani-verb-additive-causative templates)
* 20:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0f7a634e72}} (fix Hindustani verb placeholders)

=== 2023-03-05 ===
* 21:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c325634bc3}} (hindustani-verb-additive-transitive templates)
* 19:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|85cbe15d08}} (hindustani-verb-basic-transitive templates)
* 13:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|00f87cf139}} (hindustani-verb-basic-intransitive templates)

=== 2023-03-03 ===
* 20:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c310fd9d88}} (update Hindustani labels, and l10n update: tt)

=== 2023-02-27 ===
* 19:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|50aa1e2dc5}} (l10n updates: hi, hno, pa, pnb, ur)

=== 2023-02-26 ===
* 21:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1d6c0caecd}} (Hindustani non-verb templates – verbs still TBD, need more time)
* 15:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2feff85812}} (use hno translations)

=== 2023-02-22 ===
* 20:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9e667986b4}} (l10n updates: hi, ur)

=== 2023-02-14 ===
* 19:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9debac9385}} (update dependencies, especially Werkzeug 2.2.3 with two security fixes; venv rebuilt from scratch to avoid NFS issues)
* 19:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfd63ebac1}} (l10n updates: hno); also, turns out I didn’t git rebase in the last deployment, so this *actually* deploys the Danish nouns update and pl l10n update

=== 2023-02-09 ===
* 20:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2912ebfa68}} (update Danish nouns, and l10n updates: pl)

=== 2023-01-31 ===
* 19:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfaf13f447}} (update github actions; pulled without webservice restart)
* 19:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f9bf85df5f}} (l10n updates: cy)

=== 2023-01-29 ===
* 12:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3ca9650fe1}} (Danish adjectives)

=== 2023-01-09 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4857d874ce}} (l10n updates: pa)

=== 2023-01-03 ===
* 15:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c27eaec33}} (l10n updates: pl)

=== 2022-12-30 ===
* 12:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|95b9026d22}} (l10n updates: pa, zh)

=== 2022-12-28 ===
* 15:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c47032838}} (fix bulk result display when given lexeme ID)

=== 2022-12-26 ===
* 11:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b51ddc8c08}} (update Armenian noun templates)
* 11:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bdaa43aef3}} (preserve target_hash in more places)

=== 2022-12-16 ===
* 21:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4802902384}} (l10n updates: yue)

=== 2022-12-08 ===
* 19:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3f6b15c1f0}} (l10n updates: fa, gl, pl, sl)

=== 2022-12-06 ===
* 13:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|97001e468b}} (fix missing statements)
* 13:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|45a026916c}} (fix Hindko feminine noun template)

=== 2022-12-05 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4d781fb933}} (Hindko noun templates)
* 20:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2cb7ac792f}} (l10n updates: pnb)

=== 2022-12-04 ===
* 17:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|82a2272a2f}} (three new Norwegian Nynorsk noun templates)

=== 2022-11-29 ===
* 21:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b0ebae4629}} (l10n updates: el)

=== 2022-11-27 ===
* 19:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bb7cf271ae}} (l10n updates: fa)

=== 2022-11-19 ===
* 15:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|10af55574b}} (more Bokmål and Nynorsk templates)

=== 2022-11-15 ===
* 20:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5897fd06ee}} (Danish nouns fix)
* 20:23 wm-bot: &lt;lucaswerkmeister&gt; ionice -c3 zstd --rm uwsgi.log.1668543276 # 8.85%, {{Gerrit|520591680}} =&gt; {{Gerrit|46091850}} bytes)
* 20:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2b53b1199c}} (rotate uwsgi.log after 100 MiB)
* 19:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0429d7d80b}} (update Danish nouns+verbs)

=== 2022-11-10 ===
* 13:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5160edb9ca}} (l10n updates: pnb)
* 13:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|127e065522}} (NFC-normalize lemma for search)

=== 2022-11-07 ===
* 21:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0c7095c96d}} (Polish adjectives, positive only)
* 20:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c653fb07e2}} (l10n updates: es, hy, pnb)

=== 2022-11-05 ===
* 14:02 wm-bot: &lt;lucaswerkmeister&gt; git gc (.git 19M → 1.1M)
* 13:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8feb3f86d4}} (extra GitHub actions job, pulled without webservice restart)
* 12:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d38d5ba55c}} (uninstall dev dependencies in production; reduces venv size from ca. 142 MB to ca. 75 MB, or about by half)
* 12:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b7f4d4ba31}} (added test; pulled without webservice restart)
* 11:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ccecd3bb87}} (l10n updates: krc, zh)

=== 2022-10-27 ===
* 12:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|03b6dd3b71}} (l10n updates: pnb)

=== 2022-10-26 ===
* 20:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2feba604c7}} (update dependencies, use PEP 655 NotRequired)
* 19:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|55f9b203e5}} (l10n updates: sl)

=== 2022-10-23 ===
* 16:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3844a7df05}} (French verbs)

=== 2022-10-17 ===
* 19:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b098904d43}} (l10n updates: ja, pnb)

=== 2022-10-14 ===
* 18:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a829f83124}} (l10n updates: ca, hi, sh, sl)

=== 2022-10-05 ===
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|953b553968}} (translate Hebrew adjective template label)
* 18:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|93ebb772c5}} (more Spanish templates)

=== 2022-10-01 ===
* 19:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b770688eb1}} (Hebrew adjectives)
* 18:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8137259ca6}} (Flask 2.2)

=== 2022-09-23 ===
* 19:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c66922341d}} (l10n updates: ar, ku, sl)

=== 2022-09-18 ===
* 16:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8d996c1fa4}} (l10n updates: ar)

=== 2022-09-10 ===
* 18:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|609066f02b}} (README fix, pulled without webservice restart)
* 16:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|52570991cd}} (diffusion → gitlab)

=== 2022-08-29 ===
* 20:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fa8f5d87a4}} (l10n updates)

=== 2022-08-25 ===
* 14:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|019b4ecc79}} (optimize messages with unused GENDER magic word)
* 14:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dd6cb7f08b}} (l10n updates)

=== 2022-08-03 ===
* 19:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a11b6a55f6}} (l10n updates)

=== 2022-07-21 ===
* 23:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|38141487d1}} (l10n updates)

=== 2022-07-17 ===
* 17:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|238f943e8a}} (add more typing; hopefully no functional changes)

=== 2022-07-13 ===
* 20:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d5cb20368d}} (l10n updates)

=== 2022-07-02 ===
* 19:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d3e2185bbc}} (l10n updates)

=== 2022-06-29 ===
* 19:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6ac757a997}} (Igbo verbs + pronouns)

=== 2022-06-16 ===
* 21:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|466976ba49}} (l10n updates)

=== 2022-06-14 ===
* 22:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b0143851e0}} (l10n updates)

=== 2022-05-26 ===
* 20:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|24d9b273c5}} (l10n updates)

=== 2022-05-17 ===
* 19:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8cdef0cf20}} (l10n updates)

=== 2022-05-03 ===
* 20:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d8429a8740}} (l10n updates)

=== 2022-04-29 ===
* 19:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fd45333563}} (l10n updates, extra unit test)

=== 2022-04-28 ===
* 23:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|860abb205b}} (Bokmål passive verbs)

=== 2022-04-27 ===
* 20:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c92b363387}} (Mandarin templates)

=== 2022-04-25 ===
* 19:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7b5d0d7298}} (l10n updates)

=== 2022-04-22 ===
* 11:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d769b4ed8b}} (l10n updates)

=== 2022-04-20 ===
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|89a5273967}} (l10n updates)

=== 2022-04-15 ===
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; pulled {{Gerrit|24d5774c5f}} (test-only change, so no restart)
* 18:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|54a5376631}} (update German verbs)
* 16:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9a2cefe8e6}} (updated Portuguese templates)

=== 2022-04-04 ===
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|197baf2940}} (l10n updates)

=== 2022-03-30 ===
* 18:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c6001bf897}} (l10n updates; use pip-tools, includes some package updates such as Flask 2.0.2→2.1.0; clean up service.template)

=== 2022-03-19 ===
* 12:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f573b558d4}} (l10n updates)

=== 2022-03-11 ===
* 00:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d7787d7536}} (l10n updates)

=== 2022-03-05 ===
* 18:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72f2adc394}} (l10n updates)

=== 2022-02-28 ===
* 12:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|04ba7580ab}} (l10n updates)

=== 2022-02-25 ===
* 00:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1506d1a9e9}} (l10n updates)

=== 2022-02-22 ===
* 00:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1fc2f98450}} (l10n updates)

=== 2022-02-15 ===
* 13:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|56e69bad1a}} (l10n updates)

=== 2022-02-11 ===
* 23:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b4624e0bbc}} (l10n updates)

=== 2022-02-07 ===
* 13:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b3c5446831}} (l10n updates)

=== 2022-01-30 ===
* 12:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c1d6a79ed2}} (update Odia nongendered adjectives)

=== 2022-01-22 ===
* 17:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b1cc42ef84}} (Odia nouns)
* 16:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b62723fb6f}} (update Odia adverbs)

=== 2022-01-16 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|504c5481e9}} (update Spanish verbs)
* 18:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|68234bd17d}} (Odia adjectives and adverbs)

=== 2022-01-10 ===
* 18:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d1da801731}} (l10n updates)

=== 2022-01-06 ===
* 18:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|57dc392b8f}} (l10n updates)

=== 2022-01-03 ===
* 18:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aacaae3cd6}} (revert update of indefinite item ID after merge, I flipped the items)
* 15:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2eb6822ed2}} (l10n updates)
* 15:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7312514fc8}} (update indefinite item ID after merge)

=== 2022-01-01 ===
* 23:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d6110ed631}} (l10n updates)

=== 2021-12-17 ===
* 21:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|20c4392de6}} (l10n updates)

=== 2021-12-02 ===
* 23:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2a2cb9b211}} (l10n updates)

=== 2021-11-25 ===
* 21:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|baef3a16f6}} (l10n updates)

=== 2021-11-18 ===
* 13:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e001c252c5}} (l10n updates, including initial Yoruba translations)

=== 2021-11-14 ===
* 14:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c113d4dd77}} (Yoruba nouns)

=== 2021-11-08 ===
* 22:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|85719cf3ae}} (update Portuguese idioms)
* 22:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e58c43ab3e}} (Portuguese idioms quickfix)

=== 2021-11-07 ===
* 19:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|91216ed64b}} (Portuguese idioms)

=== 2021-11-06 ===
* 12:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7ef5eb34a3}} (fix Manbhumi bulk mode link)

=== 2021-11-04 ===
* 12:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d649d7a24a}} (l10n updates)

=== 2021-10-25 ===
* 19:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0f5b5de66a}} (bump startupProbe failureThreshold 3→10)
* 19:34 wm-bot: &lt;lucaswerkmeister&gt; deployment was successful after all 🤷
* 19:31 wm-bot: &lt;lucaswerkmeister&gt; belay that, the new pod hasn’t actually started properly. investigating
* 19:29 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|754342b9a3}} (language name for bn-x-Q6747180)

=== 2021-10-18 ===
* 12:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eae6c8d594}} (l10n updates)

=== 2021-10-16 ===
* 14:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1903c3d0eb}} (don’t show duplicate warning errors)
* 12:09 wm-bot: &lt;lucaswerkmeister&gt; pulled {{Gerrit|8700382f98}} (rename confusingly named deplyoment patch file) without webservice restart
* 12:04 wm-bot: &lt;lucaswerkmeister&gt; (correction on that last message, it’s a startup probe now, not a readiness probe)
* 12:03 wm-bot: &lt;lucaswerkmeister&gt; patched readiness probe into deployment again
* 12:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19fb8c90ee}} (findDuplicates fix) with full stop/start to pick up label changes

=== 2021-10-13 ===
* 23:31 wm-bot: &lt;lucaswerkmeister&gt; fully restarted webservice (stop/start) to avoid label issues
* 17:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5c87ff53c}} (remove type ignore comments) and updated dependencies, including Flask 2.0.2

=== 2021-10-11 ===
* 12:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fb32d04132}} (l10n updates)

=== 2021-10-10 ===
* 11:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bf2834c472}} (improve error handling)

=== 2021-10-04 ===
* 19:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1697521bf5}} (l10n updates)

=== 2021-09-25 ===
* 14:45 wm-bot: &lt;lucaswerkmeister&gt; removed old venv-3.7
* 13:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6f9e530018}} (mobile-friendly navbar)
* 13:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea93caf2ee}} (l10n updates)

=== 2021-09-19 ===
* 13:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c1b6e0810}} (readinessProbe → startupProbe to avoid bloating access log); deployed by adding readinessProbe: null to the patch file and patching the deployment with that

=== 2021-09-14 ===
* 20:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c36ae4154a}} (l10n updates)
* 19:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|902156ddb8}} (Croatian item ID fix)

=== 2021-09-12 ===
* 21:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4da7f64c4b}} (updates without downtime)
* 20:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f21554ab71}} (refactoring, noop)
* 15:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a4b05045d6}} (Croatian nouns)

=== 2021-09-08 ===
* 20:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2aa32a0f7f}} (l10n updates)

=== 2021-09-03 ===
* 15:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3698f0b79c}} (add passive forms to Norwegian Bokmal verbs)
* 15:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8051248b60}} (l10n updates)

=== 2021-08-30 ===
* 18:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|dfc0838301}} (l10n updates)

=== 2021-08-25 ===
* 20:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|237a5414d5}} (l10n updates)

=== 2021-08-19 ===
* 20:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bcc4c3aa63}} (l10n updates)

=== 2021-08-17 ===
* 21:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0ca42b7cdb}} (more types)
* 18:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2382c30c01}} (initial mypy setup)
* 17:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c66572938e}} (python3.9)

=== 2021-08-16 ===
* 12:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|92e5e0d70c}} (l10n updates)

=== 2021-08-14 ===
* 12:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7a1980f4e2}} (l10n updates)

=== 2021-08-11 ===
* 19:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|37acc67c90}} (l10n updates)

=== 2021-08-02 ===
* 19:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|de5ab0e740}} (l10n updates)

=== 2021-07-19 ===
* 18:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0c9f1015c0}} (work around Firefox bug)

=== 2021-07-18 ===
* 18:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fa64f7e021}} (refuse to load non-user-readable config file, guard against recurrence of [[phab:T286414|T286414]])
* 13:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61b1d0fd93}} (Igbo adjectives and fix nouns)

=== 2021-07-17 ===
* 11:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0d1f3d924e}} (load config file differently)

=== 2021-07-16 ===
* 19:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|37766a8002}} (l10n updates)

=== 2021-07-11 ===
* 20:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5dbc39eb5e}} (l10n update)
* 17:03 wm-bot: &lt;lucaswerkmeister&gt; restarted webservice to pick up 1.3 version of OAuth consumer ([[phab:T286414|T286414]])
* 13:36 wm-bot: &lt;lucaswerkmeister&gt; chmod go-rwx www/python/src/config.yaml # [[phab:T286414|T286414]]

=== 2021-07-01 ===
* 23:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ac8779515d}} (l10n updates)
* 23:37 wm-bot: &lt;lucaswerkmeister&gt; unlink ~/services.template # new version of webservice doesn’t like the symlink :(

=== 2021-06-28 ===
* 17:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|64c5584c9d}} (remove workaround for [[phab:T241422|T241422]])
* 17:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5565da07e5}} (l10n updates, especially Igbo translations)

=== 2021-06-22 ===
* 19:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c88b1962fa}} (Igbo nouns)

=== 2021-06-21 ===
* 20:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19098277f4}} (l10n updates)

=== 2021-06-20 ===
* 12:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|afc6f6f242}} (update German verbs)

=== 2021-06-19 ===
* 19:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c5b12d5dc1}} (Malayalam proper nouns)
* 19:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|05cd31e9bd}} (update Malayalam noun)

=== 2021-06-15 ===
* 20:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0b6fed0054}} (even more optional grammatical features)
* 19:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d8eadd1cae}} (more optional grammatical features)
* 18:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61a5e0fc18}} (optional grammatical features)

=== 2021-06-14 ===
* 23:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|626b73a005}} (l10n updates)
* 23:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|70efbdc1a7}} (update volitive item ID)

=== 2021-06-10 ===
* 20:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1f94df1209}} (l10n updates)

=== 2021-06-07 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|547231388b}} (add create link for duplicates in bulk mode)
* 20:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|daf88503e0}} (l10n updates)

=== 2021-06-06 ===
* 14:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2040a7497e}} (target_hash URL parameter)

=== 2021-06-05 ===
* 20:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fcf67b1016}} (improve title)

=== 2021-06-04 ===
* 23:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|16c0cd2606}} (improve batch mode results page)

=== 2021-05-31 ===
* 20:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|43a29c4369}} (replace deprecated function)
* 20:00 wm-bot: &lt;lucaswerkmeister&gt; pip upgrade (Flask 2.0.1 and other updates)
* 19:59 wm-bot: &lt;lucaswerkmeister&gt; briefly stopping tool to upgrade venv
* 18:33 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|148dafa60b}} (l10n updates)

=== 2021-05-30 ===
* 14:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3c047f6aca}} (l10n updates)

=== 2021-05-24 ===
* 18:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6ffd1a2c1b}} (update Esperanto verb)
* 16:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7d43094e56}} (l10n updates)
* 11:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e0099e68d5}} (Swedish adjective)

=== 2021-05-22 ===
* 09:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|31e85bafcf}} (l10n updates)
* 09:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|44812d4446}} (add Portuguese modal adverb)

=== 2021-05-15 ===
* 14:01 wm-bot: &lt;lucaswerkmeister&gt; tool should be back up (uwsgi.log went from 181M to 77M after moving pre-2021 data to separate files)
* 13:56 wm-bot: &lt;lucaswerkmeister&gt; briefly stopping tool (few minutes) to cycle the uwsgi.log

=== 2021-05-13 ===
* 23:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e2ceb0513}} (l10n updates)
* 14:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|67e7cf3dfb}} (rename Swedish adjective template)
* 13:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|95f40ac9d5}} (Norwegian Bokmål masculine/neuter nouns)

=== 2021-05-10 ===
* 16:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|248527544d}} (l10n updates)

=== 2021-05-09 ===
* 13:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5951b46450}} (fix lang= and dir= on index)

=== 2021-05-03 ===
* 19:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b159dd1060}} (l10n updates)

=== 2021-05-02 ===
* 11:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4c9a5f0ebf}} (duplicate check JS fixes)

=== 2021-05-01 ===
* 14:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61744950f0}} (l10n updates)

=== 2021-04-26 ===
* 19:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|abf6719d31}} (Python 3.7 fix)
* 19:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d15d0c5f2d}} (rename Dutch templates)
* 18:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|868ee95cf2}} (l10n updates)

=== 2021-04-22 ===
* 19:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8ab4ceb62a}} (l10n updates)

=== 2021-04-19 ===
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2f8f589a62}} (Swedish proper nouns)
* 20:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4effbc2a36}} (l10n updates)

=== 2021-04-17 ===
* 10:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1d10ab467e}} (fix bulk mode)

=== 2021-04-15 ===
* 19:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|051e3789a2}} (l10n updates)

=== 2021-04-14 ===
* 20:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b17ed175fe}} (move login hint up)
* 20:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0006696173}} (remove automatic login redirect)
* 12:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|30c561955f}} (login link in navbar)

=== 2021-04-12 ===
* 18:19 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e4682a00bd}} (Breton noun fixes)
* 18:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a3a81d0c4b}} (l10n updates)

=== 2021-04-09 ===
* 18:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|18bb25abd0}} (l10n updates)

=== 2021-04-05 ===
* 13:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f5439f66a2}} (l10n updates)

=== 2021-04-04 ===
* 13:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9507991400}} (Malayalam verb fix)

=== 2021-04-03 ===
* 19:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e2bc5b577}} (language code refactorings; should not result in any observable changes)
* 18:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8416f8d861}} (more Breton nouns + adverbs)
* 16:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|21201880f5}} (MarkupSafe-aware formatters; should not result in any observable changes)
* 15:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|615bba5934}} (better bulk mode errors)

=== 2021-04-02 ===
* 19:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|be73b49e29}} (better language code handling)

=== 2021-04-01 ===
* 18:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f2b128273d}} (l10n updates)

=== 2021-03-30 ===
* 21:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|7ff57d504e}} (l10n updates)

=== 2021-03-28 ===
* 19:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|43d0c29996}} (update Portuguese nouns)
* 14:16 wm-bot: &lt;lucaswerkmeister&gt; &lt;em&gt;actually&lt;/em&gt; deployed {{Gerrit|2ece3adc91}} (this time I did the &lt;code&gt;git rebase&lt;/code&gt; but forgot the &lt;code&gt;webservice restart&lt;/code&gt;, how’s that for a change)
* 13:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2ece3adc91}} (Portuguese updates)

=== 2021-03-27 ===
* 14:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1f2a6f2e17}} (replace OrderedDict with dict)
* 13:41 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4619f8cd03}} (remove duplicate template)
* 13:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9ad3addd6a}} (Malayalam verbs, and vocative case for nouns)

=== 2021-03-26 ===
* 21:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5b44b44f52}} (Malayalam verbs)
* 21:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|78a5c9a10a}} (indicate optional forms)

=== 2021-03-25 ===
* 19:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|77328e559d}} (optional forms)

=== 2021-03-24 ===
* 22:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ffa45a58b1}} (minifix)
* 19:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea6928faaa}} (clarify Norwegian Bokmål adjectives)
* 19:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|99257d861c}} (Portuguese adjectives)

=== 2021-03-23 ===
* 21:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|253aed283c}} (Latvian nouns)
* 19:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c0b2c473ff}} (add language code as ID on index page, suggested by jhsoby)

=== 2021-03-22 ===
* 21:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2e4e3dca5a}} (improved Malayalam nouns [not verbs as it says in the commit message, oops] + i18n updates)

=== 2021-03-16 ===
* 19:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|547b42f25f}} (Portuguese nouns, i18n updates)

=== 2021-03-13 ===
* 16:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f389caf9b2}} (gender i18n improvements, should be a no-op)

=== 2021-03-12 ===
* 20:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9500beeed4}} (three new translations) – should be a no-op but I didn’t want to leave it lying around without a webservice restart either
* 19:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|aa07bef3bd}} (i18n update) – also, previous SAL message mentioned {{Gerrit|712d262475}} but that’s still in &lt;code&gt;git log @..@&lt;nowiki&gt;{&lt;/nowiki&gt;u&lt;nowiki&gt;}&lt;/nowiki&gt;&lt;/code&gt;, so I think I forgot to rebase last time

=== 2021-03-10 ===
* 20:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|712d262475}} (restore logging for generic API errors)
* 19:59 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|94dfecbc2a}} (generic API error handler)

=== 2021-03-08 ===
* 14:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b7b55e1b33}} (more i18n improvements)
* 11:43 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ea7cd3ac71}} (i18n from translatewiki.net – [[phab:T272243|T272243]])

=== 2021-03-05 ===
* 22:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|109f22a415}} (Czech verbs update)

=== 2021-03-04 ===
* 21:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1435d31446}} (update Swedish translations)
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|15a24d63eb}} (minor Czech verbs improvement)

=== 2021-02-28 ===
* 17:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|369031b945}} (minifix)
* 17:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0455dc20f4}} (better OAuth error handling)

=== 2021-02-19 ===
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f66f631598}} (auth improvements)

=== 2021-02-18 ===
* 20:45 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a0ba7b84ab}} (quickfix)
* 20:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|23ccbcf6f6}} (work around [[phab:T272319|T272319]])

=== 2021-02-16 ===
* 20:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8d96af0ec2}} (add skip link)
* 19:50 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|3e716e6d6d}} (Bootstrap update)

=== 2021-02-13 ===
* 22:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|02a2edf583}} (edit summary fixes)
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7257a065e}} (code style fixes)
* 16:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e70e759d7}} (minifix)
* 13:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fb17f5e4ef}} (edit mode fix for forms with multiple representations)

=== 2021-02-11 ===
* 22:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|81166d5c17}} (reduce [[phab:T230833|T230833]] workaround / &quot;und&quot; language codes)
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8e718af67e}} (JS fix)

=== 2021-02-10 ===
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0d8279ca7f}} (&lt;script&gt; loading improvements)
* 20:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1fe3d3589e}} (prevent double submit)

=== 2021-02-04 ===
* 20:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|32b6b23f72}} (German adverbs)

=== 2021-02-01 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f4e7ba98a7}} (stop referrer-URL comparison)
* 14:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d237952e44}} (fix current_url / CSRF detection)

=== 2021-01-30 ===
* 20:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a87ce138db}} (show bulk parse errors)

=== 2021-01-28 ===
* 20:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|868bccbbe7}} (fall back to en)
* 19:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cb0855af48}} (simplify current_url)

=== 2021-01-27 ===
* 22:39 wm-bot: &lt;lucaswerkmeister&gt; deployed fixed version of test code, oops
* 22:38 wm-bot: &lt;lucaswerkmeister&gt; deployed another version of test code
* 22:26 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted test code to print current_url debug output
* 20:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1bc8d4232e}} (remove long-dead code about fixing the session cookie)
* 20:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|03255e1408}} (pop OAuth redirect target)

=== 2021-01-13 ===
* 20:28 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5725705d1}} (fix edit mode, drop form data stashing)

=== 2021-01-09 ===
* 21:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9a604413d3}} (German toponym)

=== 2021-01-07 ===
* 14:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|00d7fe313e}} (better edit links)

=== 2021-01-03 ===
* 11:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|db1e890252}} (grab cursor for draggable links)

=== 2020-12-30 ===
* 12:22 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|191518cbf9}} (edit lemma when adding first form)

=== 2020-12-23 ===
* 15:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6d8bae537b}} (Esperanto verb)
* 14:32 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|69f610af18}} (Breton noun, without mutation, collective)

=== 2020-12-22 ===
* 11:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6e1185532d}} (Basque adjective)

=== 2020-12-14 ===
* 20:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9ba55b3ad3}} (fix current_url)

=== 2020-12-13 ===
* 00:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bb0cbfc6cb}} (language code in parentheses)

=== 2020-12-12 ===
* 18:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0ec650ea2f}} (autonyms on index page)

=== 2020-12-02 ===
* 21:35 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e5291d5cda}} (more Esperanto translations)

=== 2020-11-29 ===
* 21:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|915eb4016f}} (clarify German templates)

=== 2020-11-24 ===
* 21:58 wm-bot: &lt;lucaswerkmeister&gt; undeployed debug code, I don’t remember what it was for anymore
* 21:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|59f2c38fed}} (the previously-uncommitted JS fix, now committed; some uncommitted debug code is still there)

=== 2020-11-21 ===
* 21:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1608cc4dd9}} (gender-dependent messages)

=== 2020-11-05 ===
* 19:51 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted JS fix, to be committed later if it works as intended

=== 2020-10-29 ===
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|1a150904fd}} (update Italian translations)

=== 2020-10-26 ===
* 21:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e3c4c2e664}} (Esperanto adjective)

=== 2020-10-25 ===
* 21:46 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bd4c445f02}} (edit mode fix)
* 21:11 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|782dfdabee}} (fixes for edit mode and ordia links)

=== 2020-10-24 ===
* 13:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|792db2a9f9}} (edit mode language_code parameter)

=== 2020-10-19 ===
* 20:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a7fd004ef9}} (drag’n’drop fix; submit_lexeme debug code still there)

=== 2020-10-17 ===
* 14:37 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19b5bc257a}} (more durable CSRF tokens; some uncommitted debug code to print submit_lexeme errors is still there)

=== 2020-10-08 ===
* 20:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fd8c692798}} (fix a crash; debug code still in place)

=== 2020-09-13 ===
* 08:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9f02b375f1}} (more conventient bulk mode transition; debug code still present)
* 08:17 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted extra logging for submit_lexeme errors in bulk mode

=== 2020-09-12 ===
* 12:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ce943856ed}} (fix Spanish feminine noun item ID)

=== 2020-09-08 ===
* 16:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9ac796e7aa}} (Manbhumi verbs)

=== 2020-09-06 ===
* 08:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|116e4123b0}} (fix Manbhumi duplicate search)

=== 2020-09-01 ===
* 15:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ef72c06ec8}} (Manbhumi adjectives and adverbs)

=== 2020-08-14 ===
* 19:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|13282d5404}} (Bengali verb updates)

=== 2020-08-12 ===
* 19:52 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e3291c8796}} (Bengali adverbs, other improvements)

=== 2020-08-04 ===
* 22:43 wm-bot: &lt;lucaswerkmeister&gt; &lt;em&gt;actually&lt;/em&gt; deployed {{Gerrit|39457a18ab}} (forgot to git rebase)
* 22:36 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|39457a18ab}} (Bengali adjectives and verbs)

=== 2020-07-08 ===
* 21:48 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b65c1018ff}} (translation update)

=== 2020-07-05 ===
* 22:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f29663c2b2}} (Norwegian Bokmål nouns)

=== 2020-07-04 ===
* 16:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cbf5ad6440}} (Norwegian Bokmål)

=== 2020-06-17 ===
* 23:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9b7349c602}} (update a Bengali template)

=== 2020-06-15 ===
* 20:54 wm-bot: &lt;lucaswerkmeister&gt; renamed default branch from master to main

=== 2020-06-14 ===
* 12:09 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8d5f428c3e}} (improved duplicate warning edit links)
* 10:15 wm-bot: &lt;lucaswerkmeister&gt; *actually* deployed {{Gerrit|2efe64f7e5}} (forgot to git rebase)
* 10:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2efe64f7e5}} (link edit mode in duplicate warning)

=== 2020-06-13 ===
* 21:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b42e79e6bb}} (more sections)
* 17:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cf1079fda1}} (more section improvements)
* 13:26 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c2e6d57a29}} (improved German sections)
* 11:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4cd36a71a1}} (sections in edit mode)
* 11:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4e288f0106}} (sections)
* 08:54 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bfa46d522b}} (Czech edit mode translations)

=== 2020-06-07 ===
* 20:53 wm-bot84: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9e4f3a1b65}} (two translation fixes)
* 13:35 wm-bot84: &lt;lucaswerkmeister&gt; deployed {{Gerrit|09cc2017ec}} (Bengali nouns)

=== 2020-05-24 ===
* 13:51 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c6d1c6e30}} (update Breton)

=== 2020-05-13 ===
* 22:05 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a2deb7908c}} (update past participle item ID after merge)

=== 2020-05-11 ===
* 19:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ddac27d2e2}} (translation update)

=== 2020-05-10 ===
* 22:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b797c90917}} (Breton typofix)
* 15:00 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|eac96e8493}} (Breton adjectives and other improvements)
* 11:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fc78831f8e}} (Breton nouns)

=== 2020-05-09 ===
* 19:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b4780fa832}} (drag’n’drop unmatched forms in edit mode)

=== 2020-04-25 ===
* 20:58 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|0dadbb4d4e}} (toolforge.org)

=== 2020-04-21 ===
* 21:07 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|6634452b4c}} (increase uWSGI buffer)

=== 2020-04-18 ===
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c815a210bd}} (Hebrew nouns)
* 17:34 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|33c3ac264e}} (fix english-adverb edit mode)
* 11:55 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2959ebf637}} (fix duplicates in advanced mode)

=== 2020-04-14 ===
* 20:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|44b5df2897}} (edit mode: show lemma, show conflicts, add missing statements)

=== 2020-04-13 ===
* 22:24 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2fe2118d4e}} (python3.7)
* 22:20 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ab7f751ba6}} (edit mode)

=== 2020-02-26 ===
* 00:22 wm-bot: &lt;root&gt; Migrated to 2020 Kubernetes cluster

=== 2020-01-28 ===
* 00:17 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|61fe7e59fb}} (typofix)
* 00:08 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e0e916e0a5}} (more Persian translations and RTL fixes)

=== 2020-01-27 ===
* 23:23 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|54b9e37118}} (more RTL fixes)
* 23:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|72ec256823}} (Persian nouns and verbs) [actually happened ~30mins ago, forgot to log]

=== 2020-01-15 ===
* 00:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|bc1d49c202}} (better CSRF error handling, [[phab:T242573|T242573]])

=== 2020-01-14 ===
* 00:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|242c25810b}} (clarify Spanish verbs)

=== 2020-01-12 ===
* 14:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|edcbc10ae9}} (Spanish verbs)

=== 2020-01-11 ===
* 17:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|d9619cb473}} (Danish nouns and verbs)
* 14:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|4a20b4b95e}} (Czech perfective verbs)
* 14:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8da9227b52}} (fix typos in Czech adjective template)

=== 2019-11-30 ===
* 13:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2f5a8ccc2e}} (update english-verb)

=== 2019-11-21 ===
* 22:31 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|13cf2696b9}} (reorder)
* 22:27 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|89ad1e816c}} (Basque verbs)

=== 2019-11-11 ===
* 23:30 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|cd4239904a}} (work around [[phab:T230833|T230833]])
* 21:13 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|8b53b417c1}} (fixes to Kurdish (Kurmancî))
* 17:57 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fe31bd9aa6}} (message syntax fix)

=== 2019-11-10 ===
* 19:40 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9d736fe2f6}} (Kurdish Kurmancî nouns)
* 15:56 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|29e549fe31}} (Malayalam nouns)

=== 2019-10-27 ===
* 22:15 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|2fc68fabb5}} (lexeme IDs in bulk mode)

=== 2019-10-16 ===
* 22:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b480b6d07e}} (Czech translations + adjectives with more forms)

=== 2019-10-07 ===
* 22:49 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|ce8ba2b234}} (add plural grammatical feature to Ukrainian plurale tantum forms)

=== 2019-09-30 ===
* 22:39 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|19bf4e3347}} (remove PHP_ENGINE cookie)

=== 2019-08-28 ===
* 23:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a053e9a36e}} (update Swedish translations)

=== 2019-08-22 ===
* 22:53 wm-bot: &lt;lucaswerkmeister&gt; deployed 60cf696645v (minor bulk mode improvements)
* 22:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f4fd72ab72}} (bulk mode improvements)

=== 2019-08-20 ===
* 20:21 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|938075faf2}} (bulk mode)

=== 2019-08-11 ===
* 11:25 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|09a3ac6b64}} (Swedish absolute adjectives)

=== 2019-08-02 ===
* 21:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a4d699fbcb}} (fix item ID after merge)

=== 2019-07-24 ===
* 12:18 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f0883f1ebc}} (templates API)

=== 2019-07-07 ===
* 18:47 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|50a70b3590}} (Swedish verbs)
* 13:42 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|9a148c8cc5}} (add statements when editing existing lexeme)
* 12:38 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|a8242673b9}} (use jsonify)
* 12:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|994b980655}} (CORS for duplicates API)

=== 2019-07-06 ===
* 22:03 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|b0f39bb09b}} (API to match lexemes to templates)

=== 2019-06-26 ===
* 20:14 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e74ff290cc}} (duplicates API bug fix) [actually deployed 2 hours ago, forgot to log]

=== 2019-06-24 ===
* 22:53 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|e937ff5839}} (autocapitalize=&quot;off&quot; on form)
* 22:44 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted experimental change (autocapitalize=&quot;off&quot; on form and inputs)
* 22:29 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted experimental change (autocapitalize=&quot;off&quot; on form rather than inputs)
* 22:14 wm-bot: &lt;lucaswerkmeister&gt; deployed uncommitted experimental change (autocapitalize=&quot;off&quot; on inputs)
* 21:10 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|07b05a6858}} (Portuguese verbs)

=== 2019-06-14 ===
* 19:44 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|c48127f696}} (update Russian translations)
* 00:38 wm-bot: &lt;lucaswerkmeister&gt; kubectl delete deployment lexeme-forms.purge-all-lexemes # [[phab:T225510|T225510]] done

=== 2019-06-12 ===
* 08:48 wm-bot: &lt;lucaswerkmeister&gt; kubectl create -f deployment-purge-all-lexemes.yaml # [[phab:T225510|T225510]]

=== 2019-06-10 ===
* 19:01 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|645886b3a8}} (update German translations)
* 18:16 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|846100f8d9}} (update Czech translations)
* 12:12 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|fe6cc3a79b}} (improved forms/senses message for duplicates)

=== 2019-06-09 ===
* 23:02 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|5c88de6348}} (number of forms/senses for duplicates)

=== 2019-06-08 ===
* 14:04 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|f09dfd20a1}} (Dutch nouns)
* 14:00 wm-bot: &lt;lucaswerkmeister&gt; git remote add github https://github.com/lucaswerkmeister/tool-lexeme-forms.git # work around [[phab:T224677|T224677]]
* 12:17 wm-bot: &lt;lucaswerkmeister&gt; restarted webservice after redirect loop

=== 2019-05-20 ===
* 09:06 wm-bot: &lt;lucaswerkmeister&gt; deployed {{Gerrit|496a928b67}} (switch to Python 3.5), including venv rebuild
* 08:52 wm-bot: &lt;lucaswerkmeister&gt; stopping webserver for Python 3.5 upgrade


&lt;noinclude&gt;[[Category:SAL]]&lt;/noinclude&gt;</text>
      <sha1>g7m9sh7r0a91dfo7mi59fvhjd3zpu5t</sha1>
    </revision>
  </page>
  <page>
    <title>Data Platform/Data Lake/Edits/MediaWiki history dumps</title>
    <ns>0</ns>
    <id>444886</id>
    <revision>
      <id>2433177</id>
      <parentid>2391317</parentid>
      <timestamp>2026-07-05T00:04:30Z</timestamp>
      <contributor>
        <username>Bamyers99</username>
        <id>2099</id>
      </contributor>
      <comment>/* Schema details */ +event_log_id, event_user_is_cross_wiki</comment>
      <origin>2433177</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="20663" sha1="610l7sd4jkrv8dopoa8qcfrb89q9r7y" xml:space="preserve">This page describes the data set dump of the '''denormalized revision, user and page history''' of all WMF's wikis. It is computed from the [[Analytics/Data_Lake/Edits/MediaWiki_history|MediaWiki History dataset]], lives in the [[Analytics/Systems/Cluster|Analytics Hadoop cluster]], and is downloadable from [https://dumps.wikimedia.org/other/mediawiki_history MediaWiki Dumps]. A new monthly snapshot containing all history is being produced at the beginning of each month.

= General Information =

== Content ==
This data set contains a historical record of revision (without text), user and page events of Wikimedia wikis since 2001. The data is denormalized, meaning that all events for user, page and revision are stored in the same schema. This leads to some fields being always null for some events (for instance fields about page are null in events about user). Events about users and pages have been processed to rebuild an as coherent as possible history in term of user-renames and page-moves (see [[Analytics/Systems/Cluster/Page_and_user_history_reconstruction|Page and user history reconstruction]]).  Also, some data have been preprocessed to facilitate analyses, such as edit-count per user and per page, reverting and reverted revisions and more.

== Updates ==
The updates for this data set are monthly, around the end of the month's first week. Each update contains a full dump since 2001 (the beginning of MediaWiki-time) up to the current month. The reason for this particularity is the underlying data, the MediaWiki databases. Every time a user gets renamed, a revision reverted, a page moved, etc. the existing related records in the logging table are updated accordingly. So an event triggered today may change the state of that table 10 years ago. And it turns out the logging table is the base of the MediaWiki history reconstruction process. Thus, note that incremental downloads of these dumps may generate inconsistent data. Consider using [[Event Platform/EventStreams|EventStreams]] for real time updates on MediaWiki changes ([https://stream.wikimedia.org/?doc#/Streams API docs]).

== Versioning ==
Each update receives the name of the last featured month, in YYYY-MM format. For example if the dump spans from 2001 to August 2019 (included), it will be named 2019-08 even if it will be released on the first days of September 2019. There is a folder for each available month at the root of the download URL, and for storage reasons only the last two versions are available. This shouldn't be problematic as every version contains the whole historical dataset.

== Partitioning ==
The data is organized by wiki and time range. This way it can be downloaded for a single wiki (or set of wikis). The time split is necessary because of file size reasons. There are 3 different time range splits: monthly, yearly and all-time. Very big wikis are partitioned monthly, while medium wikis are partitioned yearly, and small wikis are dumped in one single file. This way we ensure that files are not larger than ~2GB, and at the same time we prevent generating a very large number of files.
* Wikis partitioned monthly: wikidatawiki, commonswiki, enwiki.
* Wikis partitioned yearly: dewiki, frwiki, eswiki, itwiki, ruwiki, jawiki, viwiki, zhwiki, ptwiki, enwiktionary, plwiki, nlwiki, svwiki, metawiki, arwiki, shwiki, cebwiki, mgwiktionary, fawiki, frwiktionary, ukwiki, hewiki, kowiki, srwiki, trwiki, loginwiki, huwiki, cawiki, nowiki, mediawikiwiki, fiwiki, cswiki, idwiki, rowiki, enwikisource, frwikisource, ruwiktionary, dawiki, bgwiki, incubatorwiki, enwikinews, specieswiki, thwiki.
* Wikis in one single file: all the others.

== File format ==

The file format is tab-separated-value (TSV) instead of JSON in order to reduce the file sizes (JSON repeats field names for every record). Most fields of the schema are atomic (integer, string, boolean...), and a few are arrays of strings.

Some details:
* Undefined or null values are represented as an empty fields, again to make data lighter
* Encoding of string-arrays is &lt;code&gt;value1,value2,...,valueN&lt;/code&gt; with commas escaped in values.
* In text fields, carriage-returns, line-feed and tabulations are escaped with a \ to keep a valid TSV format

The files are compressed in Bzip2, for it being widely used, free software, and having a high compression rate. Note that with Bzip2, you can concatenate several compressed files and treat them as a single Bzip2 file.

== Directory structure ==
When choosing a file (or set of files) to download, the URL should look like this:

&lt;code&gt;/&amp;lt;version&amp;gt;/&amp;lt;wiki&amp;gt;/&amp;lt;version&amp;gt;.&amp;lt;wiki&amp;gt;.&amp;lt;time range&amp;gt;.tsv.bz2&lt;/code&gt;

Where
* &amp;lt;version&amp;gt; is the YYYY-MM formatted snapshot i.e. 2019-08;
* &amp;lt;wiki&amp;gt; is the wiki database name, i.e. enwiki or commonswiki;
* &amp;lt;time_range&amp;gt; is either YYYY-MM for big wikis, YYYY for medium wikis, or all-time for the rest (see partitionning above).

Examples of dump files:
* &lt;code&gt;/2019-12/wikidatawiki/2019-12.wikidatawiki.2019-05.tsv.bz2&lt;/code&gt;
* &lt;code&gt;/2019-12/ptwiki/2019-12.ptwiki.2018.tsv.bz2&lt;/code&gt;
* &lt;code&gt;/2019-12/cawikinews/2019-12.cawikinews.all-time.tsv.bz2&lt;/code&gt;

= Technical Documentation  =

'''Note''': In the documentation below, &quot;current&quot; refers to the time of the snapshot, and &quot;historical&quot; to the time of the event.  A subpage here lists answers to some frequently answered questions: [[Analytics/Data Lake/Edits/Mediawiki history dumps/FAQ]].

== Access ==

The easiest way to play with dumps is to use [[PAWS]]. See [[PAWS/PAWS examples and recipes#Dumps|these example notebooks]].

You can [[Help:Toolforge/Dumps|access the dumps through Toolforge]].  If you have a Cloud VPS instance, you can add the &lt;code&gt;mount_nfs&lt;/code&gt; role to get the &lt;code&gt;/public/dumps/public&lt;/code&gt; mount.  But you also need to enable the mount server-side, see [https://gerrit.wikimedia.org/r/c/operations/puppet/+/679790/ this patch] for example.  See [[Portal:Data_Services/Admin/Runbooks/Enable_NFS_for_a_project]] for full details.

== Schema overview ==

The dataset contains many fields (76 to be precise), but there is some structure helping in making sense of them. The fields can be divided in 5 classes:
# &lt;code&gt;event global&lt;/code&gt; fields -- They are used on every event of the dataset (&lt;code&gt;wiki_db&lt;/code&gt;, &lt;code&gt;event_entity&lt;/code&gt;, &lt;code&gt;event_type&lt;/code&gt;, &lt;code&gt;event_timestamp&lt;/code&gt;, &lt;code&gt;event_comment&lt;/code&gt;).
# &lt;code&gt;event user&lt;/code&gt; fields -- They provide information on the user having performed the event. They are set for all events in the dataset except when denormalizing user data has failed.
# &lt;code&gt;page&lt;/code&gt; fields -- They provide information about the page the event applies to. They are set for page events (&lt;code&gt;event_entity = 'page'&lt;/code&gt;) and revision events (&lt;code&gt;event_entity = 'revision'&lt;/code&gt;).
# &lt;code&gt;user&lt;/code&gt; fields -- They provide information about the user the event applies to. They are set for user events only (&lt;code&gt;event_entity = 'user'&lt;/code&gt;).
# &lt;code&gt;revision&lt;/code&gt; fields -- They provide information about the revision the event applies to. They are set for revision events only (&lt;code&gt;event_entity = 'revision'&lt;/code&gt;).

'''Note''': Except for the &lt;code&gt;event global&lt;/code&gt; class fields whose prefix is not consistent, all other have their field name prefixed with their field class.

=== Important fields: event_entity and event_type ===

Due to having user, page and revision events in the same dataset (it is said to be denormalized), filtering by &lt;code&gt;event_entity&lt;/code&gt; and possibly even &lt;code&gt;event_type&lt;/code&gt; is necessary not to mix incompatible data.

{| class=&quot;wikitable&quot;
!Entity
!Event type
!Meaning
|-
|revision
|create
|Editing a page
|-
| rowspan=&quot;6&quot; |page
|create
|Creating a page
|-
|create-page
|Page creation according to the logging table [note 1 below]
|-
|delete
|Deleting a page
|-
|move
|Changing a page's title
|-
|restore
|Undeleting a page
|-
|merge
|Merging revisions from another page [note 2 below]
|-
| rowspan=&quot;4&quot; |user
|create
|Registering of a new account
|-
|rename
|Changing the name of a user
|-
|altergroups
|Changing the groups (rights) of a user
|-
|alterblocks
|Blocking/unblocking a user
|}

* note 1: Establishing exactly when a page was created is not simple.  The logging table has a record for page creation, and we expose this in our datasets as a &quot;create-page&quot; event.  However, the first revision for some pages is *before* this logging table entry.  Therefore, we made a decision to use that event as the &quot;create&quot;.  You can follow along with our logic at [https://github.com/wikimedia/analytics-refinery-source/blob/81744162364493d65ad746ab500f0302c0080ac6/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/page/PageHistoryBuilder.scala#L778 PageHistoryBuilder #L778] and at [https://github.com/wikimedia/analytics-refinery-source/blob/81744162364493d65ad746ab500f0302c0080ac6/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/page/PageEventBuilder.scala#L180 PageEventBuilder #L180].
* note 2: we don't process merges much, and the documentation is sparse: https://www.mediawiki.org/wiki/Manual:Log_actions

== Schema details ==
Source: [https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/+/refs/heads/master/hql/mediawiki/history/create_mediawiki_history_table.hql create_mediawiki_history_table.hql] (gerrit.wikimedia.org)

{| class=&quot;wikitable&quot;
!Field class
!Field name
!Data type
!Comment
|-
|rowspan=&quot;6&quot; |Event_global
|wiki_db
|string
|enwiki, dewiki, eswiktionary, etc.
|-
|event_log_id
|bigint
|In page/user events: id of the logging table row that caused this event. NULL for revision events.
|-
|event_entity
|string
|revision, user or page
|-
|event_type
|string
|create, move, delete, etc.  Detailed explanation in the docs under #Event_types
|-
|event_timestamp
|string
|When this event ocurred
|-
|event_comment
|string
|Comment related to this event, sourced from log_comment, rev_comment, etc.
|-
|rowspan=&quot;22&quot; |Event user
|event_user_id
|bigint
|ID of the user that caused the event. Null if the user is anonymous or if from a revision where the user has been [[Mw:Manual:RevisionDelete|revision deleted]].
|-
|event_user_central_id
|bigint
|Global (central) ID of the user that caused the event
|-
|event_user_text_historical
|string
|Historical username (IP address for anonymous user) of the user that caused the event. Null for revisions where the user has been [[Mw:Manual:RevisionDelete|revision deleted]].
|-
|event_user_text
|string
|Current username of the user that caused the event. Null for anonymous users (the IP is stored in event_user_text_historical). Null for revisions where the user has been [[Mw:Manual:RevisionDelete|revision deleted]].
|-
|event_user_blocks_historical
|array&lt;string&gt;
|Historical blocks of the user that caused the event
|-
|event_user_blocks
|array&lt;string&gt;
|Current blocks of the user that caused the event
|-
|event_user_groups_historical
|array&lt;string&gt;
|Historical groups of the user that caused the event
|-
|event_user_groups
|array&lt;string&gt;
|Current groups of the user that caused the event
|-
|event_user_is_bot_by_historical
|array&lt;string&gt;
|Historical bot information of the user that caused the event, can contain values name or group
|-
|event_user_is_bot_by
|array&lt;string&gt;
|Bot information of the user that caused the event, can contain values name or group
|-
|event_user_is_created_by_self
|boolean
|Whether the event_user created their own account
|-
|event_user_is_created_by_system
|boolean
|Whether the event_user account was created by mediawiki (eg. centralauth)
|-
|event_user_is_created_by_peer
|boolean
|Whether the event_user account was created by another user
|-
|event_user_is_anonymous
|boolean
|Whether the event_user is not registered, using the old way that surfaced the IP publicly. True for revisions where the user has been [[Mw:Manual:RevisionDelete|revision deleted]], even if the user was actually registered.
|-
|event_user_is_temporary
|boolean
|Whether the event_user is not registered, using the new temporary account way. True for revisions where the user has been [[Mw:Manual:RevisionDelete|revision deleted]], even if the user was actually registered.
|-
|event_user_is_permanent
|boolean
|Whether the event_user is registered.
|-
|event_user_is_cross_wiki
|boolean
|True if the event_user is an interwiki-imported editor (usertext contains &quot;&gt;&quot;, is anonymous, and not temporary). NULL for user/page events.
|-
|event_user_registration_timestamp
|string
|Registration timestamp of the user that caused the event (from user table)
|-
|event_user_creation_timestamp
|string
|Creation timestamp of the user that caused the event (from logging table)
|-
|event_user_first_edit_timestamp
|string
|Timestamp of the first edit of the user that caused the event
|-
|event_user_revision_count
|bigint
|Number of revisions made by the &lt;code&gt;event_user&lt;/code&gt; up to the historical time in this &lt;code&gt;wiki_db&lt;/code&gt; (only available in revision-create events so far). For revision-create events, this includes the event itself.
|-
|event_user_seconds_since_previous_revision
|bigint
|In revision events: seconds elapsed since the previous revision made by the current event_user_id (only available in revision-create events so far)
|-
|rowspan=&quot;13&quot; |page
|page_id
|bigint
|In revision/page events: id of the page
|-
|page_title_historical
|string
|In revision/page events: historical title of the page
|-
|page_title
|string
|In revision/page events: current title of the page
|-
|page_namespace_historical
|int
|In revision/page events: historical namespace of the page.
|-
|page_namespace_is_content_historical
|boolean
|In revision/page events: historical namespace of the page is categorized as content
|-
|page_namespace
|int
|In revision/page events: current namespace of the page
|-
|page_namespace_is_content
|boolean
|In revision/page events: current namespace of the page is categorized as content
|-
|page_is_redirect
|boolean
|In revision/page events: whether the page is currently a redirect
|-
|page_is_deleted
|boolean
|In revision/page events: Whether the page is rebuilt from a delete event
|-
|page_creation_timestamp
|string
|In revision/page events: creation timestamp of the page
|-
|page_first_edit_timestamp
|string
|In revision/page events: timestamp of the page's first revision.  Can be before the page_creation in some restore/merge cases (see revision_is_from_before_page_creation).
|-
|page_revision_count
|bigint
|In revision/page events: Cumulative revision count per page for the current page_id (only available in revision-create events so far)
|-
|page_seconds_since_previous_revision
|bigint
|In revision/page events: seconds elapsed since the previous revision made on the current page_id (only available in revision-create events so far)
|-
|rowspan=&quot;19&quot; |user
|user_id
|bigint
|In user events: id of the user
|-
|user_central_id
|bigint
|In user events: global (central) id of the user
|-
|user_text_historical
|string
|In user events: historical username or IP address of the user
|-
|user_text
|string
|In user events: current username or IP address of the user
|-
|user_blocks_historical
|array&lt;string&gt;
|In user events: historical user blocks
|-
|user_blocks
|array&lt;string&gt;
|In user events: current user blocks
|-
|user_groups_historical
|array&lt;string&gt;
|In user events: historical user groups
|-
|user_groups
|array&lt;string&gt;
|In user events: current user groups
|-
|user_is_bot_by_historical
|array&lt;string&gt;
|In user events: Historical bot information of the user, can contain values name or group
|-
|user_is_bot_by
|array&lt;string&gt;
|In user events: Bot information of the user, can contain values name or group
|-
|user_is_created_by_self
|boolean
|In user events: whether the user created their own account
|-
|user_is_created_by_system
|boolean
|In user events: whether the user account was created by mediawiki
|-
|user_is_created_by_peer
|boolean
|In user events: whether the user account was created by another user
|-
|user_is_anonymous
|boolean
|In user events: whether the user is not registered, using the old way that surfaced the IP publicly
|-
|user_is_temporary
|boolean
|In user events: whether the user is not registered, using the new temporary account way
|-
|user_is_permanent
|boolean
|In user events: whether the user is registered
|-
|user_registration_timestamp
|string
|In user events: registration timestamp of the user.
|-
|user_creation_timestamp
|string
|In user events: Creation timestamp of the user (from logging table)
|-
|user_first_edit_timestamp
|string
|In user events: Timestamp of the first edit of the user
|-
|rowspan=&quot;18&quot; |revision
|revision_id
|bigint
|In revision events: id of the revision
|-
|revision_parent_id
|bigint
|In revision events: id of the parent revision
|-
|revision_minor_edit
|boolean
|In revision events: whether it is a minor edit or not
|-
|revision_deleted_parts
|array&lt;string&gt;
|In revision events: Deleted parts of the revision, can contain values text, comment and user
|-
|revision_deleted_parts_are_suppressed
|boolean
|In revision events: Whether the deleted parts are deleted to admin as well (visible only by stewards)
|-
|revision_text_bytes
|bigint
|In revision events: number of bytes of revision
|-
|revision_text_bytes_diff
|bigint
|In revision events: change in bytes relative to parent revision (can be negative).
|-
|revision_text_sha1
|string
|In revision events: sha1 hash of the revision
|-
|revision_content_model
|string
|In revision events: content model of revision
|-
|revision_content_format
|string
|In revision events: content format of revision
|-
|revision_is_deleted_by_page_deletion
|boolean
|In revision events: whether this revision has been deleted (moved to archive table)
|-
|revision_deleted_by_page_deletion_timestamp
|string
|In revision events: the timestamp when the revision was deleted
|-
|revision_is_identity_reverted
|boolean
|In revision events: whether this revision was reverted by another future revision
|-
|revision_first_identity_reverting_revision_id
|bigint
|In revision events: id of the revision that reverted this revision
|-
|revision_seconds_to_identity_revert
|bigint
|In revision events: seconds elapsed between revision posting and its revert (if there was one)
|-
|revision_is_identity_revert
|boolean
|In revision events: whether this revision reverts other revisions
|-
|revision_is_from_before_page_creation
|boolean
|In revision events: True if the revision timestamp is before the page creation (can happen with restore events)
|-
|revision_tags
|array&lt;string&gt;
|In revision events: Tags associated to the revision
|}

== Code examples ==

* [[Data Platform/Data Lake/Edits/MediaWiki history dumps/Scala spark examples|Scala spark examples]]

* [[Data Platform/Data Lake/Edits/MediaWiki history dumps/Python spark examples|Python spark examples]]

* [[Data Platform/Data Lake/Edits/Mediawiki history dumps/Python Pandas examples|Python Pandas examples]]

* [[Data Platform/Data Lake/Edits/Mediawiki history dumps/Python Dask examples|Python Dask examples]]

== Pipeline ==

# &lt;code&gt;mediawiki_history_dumps&lt;/code&gt; [https://airflow-analytics.wikimedia.org/dags/mediawiki_history_dumps/grid Airflow DAG] waits on monthly &lt;code&gt;snapshot&lt;/code&gt; of &lt;code&gt;wmf.mediawiki_history&lt;/code&gt;.
# DAG then transforms &lt;code&gt;wmf.mediawiki_history&lt;/code&gt; with the Spark job [https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/MediawikiHistoryDumper.scala MediawikiHistoryDumper.scala], making the dump available in HDFS at &lt;code&gt;/wmf/data/archive/mediawiki/history&lt;/code&gt;.
# Separately, a daily [https://github.com/wikimedia/operations-puppet/blob/1fadb54de84f1f0b1e0e1d2d5279bc702a7fccaf/modules/dumps/manifests/web/fetches/stats.pp#L47-L59 hdfs-rsync job] syncs the last two dumps to the [[Dumps/Dump servers|Dump servers]] to make them available publicly at https://dumps.wikimedia.org/other/mediawiki_history/.
# Finally, a [https://github.com/wikimedia/operations-puppet/blob/1fadb54de84f1f0b1e0e1d2d5279bc702a7fccaf/modules/profile/manifests/analytics/refinery/job/data_purge.pp#L226-L236 clean-up job] makes sure dumps older than 180 days are removed from HDFS.

= Changes and known problems =

====== MediaWiki history dump files ======
{| class=&quot;wikitable&quot;
!Date
!PhabTask
!Snapshot version
!Details
|-
|2020-01
|
|2019-12
|Initial release
|}

====== Upstream mediawiki_history data lake table ======
See: [[Data Platform/Data Lake/Edits/MediaWiki history#Changes and known problems]]
[[Category:Edits data]]
[[Category:Data platform]]</text>
      <sha1>610l7sd4jkrv8dopoa8qcfrb89q9r7y</sha1>
    </revision>
  </page>
  <page>
    <title>Map of database maintenance</title>
    <ns>0</ns>
    <id>449160</id>
    <revision>
      <id>2433176</id>
      <parentid>2433160</parentid>
      <timestamp>2026-07-05T00:02:21Z</timestamp>
      <contributor>
        <username>Dexbot</username>
        <id>30554</id>
      </contributor>
      <minor />
      <comment>Bot: Updating the report</comment>
      <origin>2433176</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="1524" sha1="sof6dme4f60qimy53cx80ev4wll1jvr" xml:space="preserve">{{/Header}}
== Today (2026-07-05) ==
== Yesterday (2026-07-04) ==
== Last seven days ==
{| class=&quot;wikitable&quot;
|+ eqiad
|-
! Section !! Work
|-
| es7 || [[phab:T430765|Switchover es7 master (es1039 -&amp;gt; es1035) (T430765)]] (ladsgroup)
|-
| s3 || 
* [[phab:T426633|Login (T426633)]] (fceratto)
* [[phab:T430610|Switchover s3 master (db1223 -&amp;gt; db1189) (T430610)]] (marostegui)
|-
| s4 || [[phab:T430817|Switchover s4 master (db1244 -&amp;gt; db1160) (T430817)]] (cwilliams)
|-
| s5 || 
* [[phab:T426633|Login (T426633)]] (fceratto)
* [[phab:T430540|Switchover s5 master (db1230 -&amp;gt; db1210) (T430540)]] (marostegui)
|-
|}
{| class=&quot;wikitable&quot;
|+ codfw
|-
! Section !! Work
|-
| s2 || 
* [[phab:T426633|Login (T426633)]] (fceratto)
* [[phab:T430624|Switchover s2 master (db2204 -&amp;gt; db2207) (T430624)]] (fceratto)
|-
| s3 || 
* [[phab:T410589|Optimize all core tables, late 2025 (T410589)]] (ladsgroup)
* [[phab:T426633|Login (T426633)]] (fceratto)
* [[phab:T430912|Switchover s3 master (db2205 -&amp;gt; db2209) (T430912)]] (fceratto)
|-
| s4 || [[phab:T430127|Switchover s4 master (db2240 -&amp;gt; db2179) (T430127)]] (cwilliams)
|-
| s5 || 
* [[phab:T426633|Login (T426633)]] (fceratto)
* [[phab:T430923|Switchover s5 master (db2213 -&amp;gt; db2192) (T430923)]] (fceratto)
|-
| s6 || [[phab:T430814|Switchover s6 master (db2214 -&amp;gt; db2229) (T430814)]] (fceratto)
|-
| s7 || 
* [[phab:T426633|Login (T426633)]] (fceratto)
* [[phab:T430826|Switchover s7 master (db2220 -&amp;gt; db2159) (T430826)]] (fceratto)
|-
|}


[[Category:MariaDB]]</text>
      <sha1>sof6dme4f60qimy53cx80ev4wll1jvr</sha1>
    </revision>
  </page>
  <page>
    <title>Performance/Multi-DC MediaWiki</title>
    <ns>0</ns>
    <id>451565</id>
    <revision>
      <id>2433175</id>
      <parentid>2345735</parentid>
      <timestamp>2026-07-04T18:58:27Z</timestamp>
      <contributor>
        <username>Krinkle</username>
        <id>44</id>
      </contributor>
      <comment>/* CDN purging */</comment>
      <origin>2433175</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="18612" sha1="auhj87wsssnd8t35zyux81jqfb5quff" xml:space="preserve">'''Multi-DC MediaWiki''' (also known as, ''Active-active MediaWiki'') is a cross-cutting project driven by the [[Performance|Performance Team]] to give MediaWiki the ability to serve read requests from multiple datacenters.

Historically, Wikimedia Foundation served requests to MediaWiki from its primary datacenter only, where the secondary data center was effectively a cold stand-by exclusively for disaster recovery. By deploying and actively serving MediaWiki from two (or more) datacenters during normal operations, we achieve higher resilience in case of network loss or data center failure, remove most switchover cost and complexity, and ease regular maintenance.

Throughout the project, Multi-DC initiatives have brought about performance, availability, and resiliency improvements across the MediaWiki codebase and at every level of our infrastructure. These gains were effectively and advantageous even to the then-single DC operations. These gains were often due to restructuring business logic and implementing async and eventual-consistency solutions.

The project's final switch ([[phab:T279664|T279664]]) improved page load speed by reducing latencies to geographies west of Texas, USA ([[Codfw data center]]), including for Asia and US West coast. This impacted all logged-in traffic as well as cache misses from logged-out traffic. The project brings a promise of future performance potential through serving more traffic from nearby DCs.

== Initial work (2014-2016) ==

=== RFC ===
The project was formalised via the [[mw:Requests_for_comment/Multi_datacenter_strategy_for_MediaWiki|Multi-DC strategy RFC]] in 2015, led by Aaron Schulz (Performance Team).

=== JobQueue ===
Changes led by Aaron in collaboration with Ori (Performance Team), including:

* Develop JobQueueAggregator in MediaWiki, using Memcached and Redis for coordination.
* Develop new &quot;jobrunner&quot; microservice in 2014.
* Develop the &quot;jobchron&quot; service in 2015.

* Infrastructure changes to replicate the queue cross-dc by default, through Redis.
* New interfaces in MediaWiki to support independent queuing of local jobs in either DC, which eventually replicate to and execute in the primary DC. 

See also [[History of job queue runners at WMF]].

=== WANObjectCache ===
Designed and implemented by Aaron. Read more about this in:

* [https://techblog.wikimedia.org/2019/02/14/perf-matters-at-wikipedia-in-2015/ Perf Matters 2015 § Hello, WANObjectCache]
* [https://techblog.wikimedia.org/2022/12/08/perf-matters-at-wikipedia-2016/#one-step-closer-to-multi-dc Perf Matters 2016 § One step closer to Multi DC]
* [[phab:T88492|T88492: Devise memcached strategy for multi-DC mediawiki]]
=== MariaDB lag indication ===
The default mechanism in MySQL to let web applications measure database replication lag, is to fetch the &lt;code&gt;Seconds_behind_master&lt;/code&gt; value using &quot;SHOW STATUS&quot; queries. This works well for small sites, and is what MediaWiki does by default as well. At our scale, however, this can be inaccurate or unreliable. For example, DBAs tend to prefer a replication topology where a chain is used rather than all replicas sourcing a single primary. This eases cross-dc switchovers and day to day intra-dc maintenance. A number of other issues and use cases are detailed at [[phab:T111266|T111266]].

To mitigate this we adopted the [[MariaDB/pt-heartbeat|pt-heartbeat]] service. Deployment led by Jaime Crespo (SRE Data Persistence) with the MediaWiki client (within the Rdbms library) written by Aaron.

=== Media storage ===
See also [[Media storage#History]]. MediaWiki's FileBackend abstraction layers was developed in 2010-2012 by Aaron to facilitate WMF's transition from NFS to Swift, and was extended and excercised further during the migration from the [[Pmtpa]] data center to, our present-day primary DC, Eqiad.

In 2015, further work took place on SwiftFileBackend and FileBackendMultiwrite in order to not rely on periodic out-of-bound replication but rather have MediaWiki directly write to both data centers ([[phab:T91869|T91869]], [[phab:T112708|T112708]], [[phab:T89184|T89184]]). 

In 2016, the encryption for cross-dc traffic between the two Swift clusters was finalized by SRE, involving Nginx as TLS proxy. [[phab:T127455|T127455]]

=== Search ===
Devise strategy for near-realtime Elasticsearch replication so that search suggestions and search queries work independently. Led by Erik Bernhardson (Search Platform). [[phab:T91870|T91870]]

== Incremental progress ==

=== DC independence ===
Throughout the codebase, individual components had to be improved, refactored, or even rewritten entirely to be independent of a (slow and possibly overloaded) connection to the primary DC. This largely took place between 2014 and 2017, led by Aaron Schulz, tracked under [[phab:T88445|T88445]] and [[phab:T92357|T92357]]. Including:

* Introduce a TransactionProfiler in the Rdbms library, this automatically brings performance issues to the attention of developers. In particular, to identify unexpected primary DB writes and reads during &quot;GET&quot; requests. [[phab:T137326|T137326]]

* Reduce write() calls in MediaWiki's SessionStore.
* Introduce &quot;sticky-DC&quot; cookie, set during requests that save or change data (e.g. edits, changing user preference, etc.)
* Convert the localisation cache and MessageCache to WANObjectCache. ([[phab:T99208|T99208]])
* Refactor Wikibase extension to avoid eager connections to the primary DB.
* Refactor Flow extension to reduce database queries during GET requests. 
* Revise Flow caching strategy from caching primary data to caching data from DB replicas instead. More about this in [https://techblog.wikimedia.org/2022/12/08/perf-matters-at-wikipedia-2016/#one-step-closer-to-multi-dc Perf Matters 2016 § One step closer to Multi DC].

=== Mcrouter ===
The relay interface exposed by WANObjectCache was fulfilled in 2016 by [[Mcrouter]], which replaced WMF's prior Twemproxy and Nutcracker infrastructure ([[phab:T132317|T132317]]). Configuration and deployment led by Giuseppe Lavagetto (SRE Service Ops) in 2018 ([[phab:T192370|T192370]]).

=== CDN purging ===
The foundation's CDN by its very nature has always operated from multiple DCs and so MediaWiki has already supported sending purges across multiple DCs. Nonetheless, there have been long-standing problems with the reliability or effectiveness of CDN purges. These are detailed and tracked under [[phab:T133821|T133821]], and was worked on from 2015 to 2020, including:

* In 2015, Aaron Schulz introduced the [[mw:Manual:$wgCdnReboundPurgeDelay|$wgCdnReboundPurgeDelay]] feature which emits a second cache purge after a configurable delay ([[phab:T113192|T113192]]). When one of the database replicas is lagged (up to 5 seconds was tolerated before the replica is automatically depooled), a purge can be reliable and yet ineffective because when we repopulate the CDN cache with a fresh pageview generated by MediaWiki, it may consult that lagged replica and essentially re-create the stale page we just purged. The rebound is issued after 11 seconds.
* From 2017-2018, incremental improvements to [[Multicast HTCP purging]] took place led by Emanuele Rocca and Brandon Black (SRE Traffic). This included chaining the purge queues such that the backend Varnish is purged before the frontend Varnish, as well as purging Varnish backends in the primary DC before Varnish backends in the caching DCs, to avoid race conditions where a higher tier may repopulate the cache with a stale copy from a lower tier. See also [[CDN/History#2015-2019]].
* In 2019, after replacing Varnish backends with ATS backends, the second tier chain (i.e. &quot;origin shield&quot;) was removed. Each CDN edge server still has two layers (coloated Varnish frontend and ATS backend), but after that an edge server will fetch directly from MediaWiki app servers in the primary DC, instead of going through a second backend cache tier in the primary DC. ([[CDN/History#2019]]).
* In 2020, the [[Purged]] service was developed and deployed to solve the reliability issue long-term by using Kafka instead of HTCP (multicast UPD).

=== Thumbnail serving ===
Thumbor and storing thumbnails in Swft in both DCs. Carried out in 2018 with traffic routing implemented by Filippo (SRE Infrastructure Foundations), and application layer work by Gilles (Performance Team). [[phab:T201858|T201858]]

=== Media originals serving ===
In 2019, we began to serve originals from upload.wikimedia.org from multiple data centers, implemented by Alexandros Kosiaris and Filippo (SRE). [[phab:T204245|T204245]]

== Remaining work (2020-2022) ==
{{Tracked|T270223}}
Aaron Schulz has driven the effort of improving, upgrading, and porting the various production systems around MediaWiki to work in an active-active context with multiple datacenters serving MediaWiki web requests. You can see [[phab:T88445|the history of subtasks on Phabricator]].

This document focuses on remaining work left as of December 2020 – the major blockers left before enabling the active-active serving of MediaWiki.

=== ChronologyProtector ===
[[mw:Manual:ChronologyProtector|ChronologyProtector]] is the system ensuring that editors see the result of their own actions in subsequent interactions.

The remaining work is deciding where and how to store the data going forward, to deploy any infra and software changes as needed, and to enable these.

* Lead: Performance Team (Timo).
* In collaboration with [[SRE/Service Operations|SRE Service Operations]].
* Task: [[phab:T254634|T254634]]

Updates:

* [[phab:T254634#6433868|September 2020]]: an architectural solution has been decided on and the Performance Team, in collaboration with [[SRE/Service Operations|SRE ServiceOps]], will migrate ChronologyProtector to a new data storage (either Memcached or Redis), during Oct-Dec 2020 (FY 2020-2021 Q2).
*[[phab:T254634#6832141|February 2021]]: code simplification and backend configuration for Multi-DC ChronologyProtector have been implemented and deployed to production for all wikis.
*March 2021: Documented [https://doc.wikimedia.org/mediawiki-core/master/php/classWikimedia_1_1Rdbms_1_1ChronologyProtector.html#ChronologyProtector-storage-requirements CP store requirements] for third-parties.
*March 2021: Task closed.
*{{Done}}

=== Session storage ===
The session store holds temporary data required for authenticating and authorization procedures such as logging in, creating accounts, and security checks before actions such as editing pages.

The older data storage system has various short-comings beyond mere incompatibility with a multi-DC operation. Even in our current single-DC deployment the annual switchovers are cumbersome, and a replacement has been underway for some time.

The remaining work is to finish the the data storage migration from Redis (non-replicated) to [[Kask]] (Cassandra-based). 

* Lead: &lt;s&gt;[[mw:Core Platform Team|Core Platform Team]], &lt;/s&gt; Performance Team (Timo, Tim).
* Past tasks: [[phab:T206010|RFC T206010]], [[phab:T206016|T206016]].
* Current tasks: [[phab:T270225|T270225]] (core logic).

Updates:

* 2018-2020 ([[phab:T206016|T206016]]): Develop and deploy Kask, gradually roll out to all Beta and production wikis. 
* Dec 2020: Performance Team realize that requirements appear unmet, citing multiple unresolved &quot;TODOs&quot; in the code for primary requirements, internally inconsistent claims about service interface. [[phab:T270225|T270225]]
* Jan 2021: CPT triages task from Inbox.
* Feb 2021: CPT moves task to &quot;''Platform Engineering Roadmap &gt; Later''&quot;. 
* March 2021: Future optimisation identified by Tim (two-level session storage) - [[phab:T277834|T277834]].
* July 2022: Performance helps with [[phab:T270225|T270225]] within the limited scope of completing Multi-DC needs.
* July 2022: Fulfilled unresolved TODOs at [[phab:T270225|T270225]].
* July 2022: Straighten out internally inconsistent interface guarantees at [[phab:T270225|T270225]].
* August 2022: Task closed, {{Done}}.

=== CentralAuth storage ===
A special kind of session storage for the central login system and cross-wiki &quot;auto login&quot; and &quot;stay logged in&quot; mechanism.

The last part of that work, [[phab:T254422|migrating]] [[mw:Extension:CentralAuth|CentralAuth]] sessions, is currently scheduled for completion in Oct-Dec 2020 (2020-2021 Q2).

* Lead:Performance Team (Tim), &lt;s&gt;[[mw:Core Platform Team|Core Platform Team]] (Bill Pirkle)&lt;/s&gt;.
* Task: [[phab:T267270|T267270]]

Updates:

* Nov 2020: Initial assessment done by CPT.
* Jan 2021: Assessment concluded.
* Feb 2021: Assessment re-opened.
* Jul 2022: Decided on Kask-sessions (Cassandra) as backend. Should not be separate from core sessions. TTL mismatch considered a bug and also fixed by Tim. [[phab:T313496|T313496]]
* Jul 2022: {{Done}}

=== Main Stash store ===
The [[Redis|Redis cluster]] previously used for session storage is also host to other miscellaneous application data through the [[mw:Manual:Caching#Main stash|Main Stash]] interface. This has different needs than session storage which become more prominent in a multi-DC deployment which make it unsuitable for Cassandra/Kask.

The remaining work is to survey the consumers and needs of Main Stash, decide how to accomodate them going forward. E.g. would it help if we migrated some of its consumers elsewhere and have a simpler replacement for the rest? Also: carry out any software and infra changes as needed.

* Lead: [[SRE/Data Persistence|SRE Data Persistence Team]] (Manuel).
* In collaboration with [[Performance|Performance Team]] (Tim, Aaron)
* Task: [[phab:T212129|T212129]]

Updates:

* Sept 2019: [https://docs.google.com/document/d/1tX8ekiYb3xYgpNJsmA1SiKqzkWc0F-_E4SGx6BI72vA/edit Audit all current MainStash usage (Google Doc)].
* June 2020: The plan is to move this data to a new small MariaDB cluster. This project requires fixing &quot;makeGlobalKey&quot;&quot; in SqlBagOStuff, and new hardware. This is being procured and set up in Q2 2020-2021 by the Data Persistence Team. The Performance Team will take care of migrating the Main Stash as soon as the new database cluster is available, i.e. between Oct 2020 and Mar 2021 (FY 2020-2021 Q2 or Q3).
* July 2020: SqlBagOStuff now supports makeGlobalKey and can work with separate DB connections outside the local wiki. - [[phab:T229062|T229062]]
* Sep 2020: Hardware procurement submitted. Oct 2020: Procurement approved as part of larger order. Dec 2020: Hardware arrived. - [[phab:T264584|T264584]]
* Jan 2021: Hardware racked and being provisioned. - [[phab:T269324|T269324]] 
*Feb 2021: MySQL service online and replication configured. - [[phab:T269324|T269324]] 
*June 2022: Test config in production.
*June 2022: Enable on all wikis.
*{{Done}} 

=== MariaDB cross-datacenter secure writes ===
MediaWiki being active-active means that writes still only go to the primary datacenter, however a fallback is required for edge cases where a write is attempted in a secondary datacenter. In order to preserve our users' privacy, [[phab:T134809|writes need to be sent encrypted across datacenters]]. Multiple solutions are being considered, but a decision has yet to be made on which one will be implemented. This work will be a collaboration between the SRE Data Persistence Team and the Performance Team. We hope for it to happen during fiscal year 2020-2021.

* Lead: [[SRE/Data Persistence|SRE Data Persistence Team]] (Manuel).
* In collaboration with [[Performance|Performance Team]] (Tim, Timo, Aaron).
* Task: [[phab:T134809|T134809]]

Updates:

* July 2020: Potential solutions evoked so far: Connect with TLS to MariaDB from PHP directly, ProxySQL, dumb TCP tunnel, Envoy as TCP tunnel?, HAProxy in TCP mode.
* Dec 2020: Leaning toward a tunnel approach, ProxySQL would take too long to set up and test from scratch.
* May 2022: Decision is reached, led by Tim. TLS connection to be established directly from MediaWiki without additional proxies or tunnels.
* May 2022: Configuration written.
* June 2022: MariaDB-TLS tested and enabled for all wikis.
* {{Done}}

=== ResourceLoader file dependency store ===
Currently written to a core wiki table using a primary DB connection, must be structured such that writes are done within a secondary DC and then replicated. The plain is to migrate it toward the Main Stash instead.

* Lead: Performance Team (Timo, Aaron).
* Task: [[phab:T113916|T113916]]

Updates:

* July 2019: Implement DepStore abstraction, decoupled from using primary DB, and now includes a KeyValue implementation that supports Main Stash.
* May 2020: Rolled out to Beta Cluster and mediawiki.org.
* June 2022: MainStashDB went live. Roll out to group0 wikis.
* July 2022: Gradually rolled out to all wikis (details on task).
*{{Done}}

=== CDN routing  ===
Remaining work is to agree on the MW requirements, and then write, test and deploy the traffic routing configuration.

* Lead:  Performance Team (Tim, Aaron, Timo).
* In collaboration with SRE Traffic and SRE Service Ops.
* Task: [[phab:T91820|T91820 (implement CDN switch)]],  [[phab:T279664|T279664 (plan gradual rollout)]] 

Updates:

* May 2020: Aaron and Timo have thought through all relevant scenarioes and drafted the requirements at [[phab:T91820|T91820]].
* June 2020: Audit confirms that relevant routing cookies and headers are in place on the MW side.
* May 2022: Traffic routing logic being developed by Tim.
* June 2022: ATS routing logic deployed to Beta and prod, no-op but enabled.
* Sept 2022: Gradually roll out based on a percentage of cache-miss traffic between ATS and appserver-ro (MediaWiki servers).
* {{Done}}

== Further reading ==
* [https://techblog.wikimedia.org/2019/02/14/perf-matters-at-wikipedia-in-2015/ ''Perf Matters 2015: Hello, WANObjectCache''] (Timo Tijhof), techblog.wikimedia.org.
* [https://techblog.wikimedia.org/2022/12/08/perf-matters-at-wikipedia-2016/#one-step-closer-to-multi-dc ''Perf Matters 2016: One step closer to Multi DC''] (Timo Tijhof), techblog.wikimedia.org
* [https://techblog.wikimedia.org/2023/05/08/around-the-world-how-wikipedia-became-a-multi-datacenter-deployment/ ''Around the world: How Wikipedia became a multi-datacenter deployment''] (Aaron Schulz), techblog.wikimedia.org, 2023.
== History ==

* For notes from 2015-2016, see [[mw:Requests for comment/Multi datacenter strategy for MediaWiki/Progress|Multi datacenter strategy for MediaWiki/Progress]].
* [[mw:Requests for comment/Multi datacenter strategy for MediaWiki|Multi-DC strategy RFC]] from 2015.
* [[mw:WMF Projects/Master &amp; Secondary Datacenters|Multi-DC project pitch]] from March 2016.

[[Category:Wikimedia Performance|Multi-DC]]</text>
      <sha1>auhj87wsssnd8t35zyux81jqfb5quff</sha1>
    </revision>
  </page>
  <page>
    <title>User:Ameisenigel~labswiki</title>
    <ns>2</ns>
    <id>457864</id>
    <revision>
      <id>2433181</id>
      <parentid>2426745</parentid>
      <timestamp>2026-07-05T02:30:39Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>Interwiki redirect to evade bot</comment>
      <origin>2433181</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="39" sha1="22fzz8ixqyw0qnqucktbxv5sy2wptk8" xml:space="preserve">{{Interwiki redirect|User:Ameisenigel}}</text>
      <sha1>22fzz8ixqyw0qnqucktbxv5sy2wptk8</sha1>
    </revision>
  </page>
  <page>
    <title>Technical partnerships</title>
    <ns>0</ns>
    <id>458581</id>
    <revision>
      <id>2433189</id>
      <parentid>2285440</parentid>
      <timestamp>2026-07-05T02:50:53Z</timestamp>
      <contributor>
        <username>Mryclai</username>
        <id>57258</id>
      </contributor>
      <minor />
      <comment>IG:Yc.lai.2088</comment>
      <origin>2433189</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="175" sha1="dvl1nft7yljlz05p78uuw02d6id7beo" xml:space="preserve">[[Git-buildpackage|To]] explore options for technical partnerships with the [[meta:Wikimedia Foundation|Wikimedia Foundation]], email {{email|techpartnerships|wikimedia.org}}.</text>
      <sha1>dvl1nft7yljlz05p78uuw02d6id7beo</sha1>
    </revision>
  </page>
  <page>
    <title>Help:Toolforge/Managing and sharing files</title>
    <ns>12</ns>
    <id>460342</id>
    <revision>
      <id>2433183</id>
      <parentid>2428379</parentid>
      <timestamp>2026-07-05T02:37:08Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>lang=&quot;bash&quot; sprinkling</comment>
      <origin>2433183</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="11717" sha1="af3jw8sl7xdeszkr8et516i0trsykwt" xml:space="preserve">{{Toolforge nav}}
This guide explains how to store, transfer, share, and manage files in Toolforge. It covers common tasks such as choosing the appropriate storage location, protecting sensitive files with the correct permissions, transferring files to tool accounts, and sharing files with other maintainers.

Use this guide when you need to work with files as a Toolforge tool maintainer.

== Before you start ==
This guide assumes that you:

* Have a Toolforge shell account.
* Are a maintainer of the tool account you want to work with.
* Can connect to Toolforge using SSH.
* Have a basic understanding of Linux command-line tools and file permissions.

Some examples use placeholder names such as toolaccount, toolname, and yourshellaccountname. Replace these with the appropriate values for your environment.

=== Understanding Toolforge storage ===
Toolforge provides several storage locations for tool code, data, and files. These locations differ in who can access them and how they are intended to be used.

As a tool maintainer, it is important to understand which locations are shared, which are private, and which are publicly accessible. Choosing the appropriate storage location helps ensure that files are available to the right users and services while protecting sensitive information.

=== Choosing where to store files ===
Before transferring or creating files in Toolforge, choose a storage location based on who needs access to the files and how they will be used.

Toolforge provides several storage locations with different access controls and intended uses.
{| class=&quot;wikitable&quot;
!Location
!Access
!Intended use
|-
|/data/project/toolname
|Shared among tool maintainers
|Use this location for tool source code, configuration files, and persistent data that must be available to the tool and its maintainers.
|-
|/data/scratch
|Shared across Toolforge users
|Use this location for temporary files, caches, and other data that can be recreated if needed.
|-
|/home
|Personal workspace
|Use it for shell configuration files, small scripts, and work in progress that is not yet part of a tool.
|-
|public_html
|Publicly accessible
|Store files here only if they are intended to be served by a web service and viewed by others.
|-
|/public/dumps
|Public read-only asscess
|Use this location when your tool needs to read dump data without maintaining a local copy.
|-
|/data/project/shared/mediawiki
|Read-only checkout of MediaWiki repositories
|Use this location for code inspection, searching across repositories, and development tasks that require access to MediaWiki source code.
|}

=== Storage recommendations ===

* Use /data/project/toolname for files that are required by your tool.
* Use /data/scratch only for temporary or reproducible data.
* Do not store passwords, API keys, or other sensitive information in shared locations.
* Do not store databases, log files, or frequently modified data on shared NFS storage.
* Store source code in version control rather than relying on shared storage as the primary copy.

== Managing file permissions ==
Toolforge is a shared environment. Files that you create may be accessible to other users, so sensitive files such as passwords, API keys, or configuration files should have restricted permissions.

To create a new private file that is only readable and writable by its owner:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
install -m 600 /dev/null FILE_NAME
&lt;/syntaxhighlight&gt;Choose file permissions carefully before storing sensitive information.

== Transferring files ==
Toolforge supports several ways to move and work with files, depending on your workflow. Common methods include scp for copying files, sshfs for local editing, and rsync for synchronizing directories.

=== Using scp ===
You can transfer files from your local machine to Toolforge using ''scp''.

For example, to make a file available to a tool account:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
cp somefile ~tools.toolaccount/
&lt;/syntaxhighlight&gt;After copying a file into a tool account's directory, the tool account must take ownership of the file before it can use it.

=== Taking ownership of files ===
The &lt;code&gt;take&lt;/code&gt; command changes ownership of files and directories to the calling tool account.

Before using &lt;code&gt;take&lt;/code&gt;, the tool account must already have access to the file through either ownership or group permissions.

Become the tool account:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
become toolaccount
&lt;/syntaxhighlight&gt;Take ownership of the file:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
take FILE
&lt;/syntaxhighlight&gt;

=== Working with files locally using sshfs ===
If you prefer to edit files using a local IDE such as Visual Studio Code or Eclipse, you can mount your Toolforge directory using sshfs instead of editing files directly from the command line.

Create a local mount point and mount the remote directory:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
sshfs yourshellaccountname@login.toolforge.org:/data/project/toolname ~/remote
&lt;/syntaxhighlight&gt;Unmount the directory when finished:&lt;syntaxhighlight lang=&quot;bash&quot;
umount ~/remote
&lt;/syntaxhighlight&gt;If a normal unmount fails, you can force an unmount:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
fusermount -zu ~/remote
&lt;/syntaxhighlight&gt;

=== Synchronizing files with rsync ===
For larger or repeated file transfers, rsync can efficiently synchronize local and Toolforge directories.

A basic example:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
rsync --archive yourLocalDirectory 
login.toolforge.org:/home/yourshellaccountname/destinationDirectory
&lt;/syntaxhighlight&gt;To synchronize files directly into a tool account directory, configure rsync to use the appropriate tool account and destination path.

=== Common problems accessing transferred files ===

==== The tool cannot access transferred files ====
Check that:

* The file has been copied to the correct location.
* The tool account owns the file.
* File permissions allow the required access.

If necessary, become the tool account and run:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
take FILE
&lt;/syntaxhighlight&gt;

== Sharing files ==
Toolforge provides several ways to share files and code with other maintainers. Choose a method based on what you are sharing and how it will be maintained.

=== Share files using a tool account ===
Files stored in a tool account's project directory can be accessed by maintainers of that tool. This is the preferred way to share configuration files, generated data, and other files that are required by the tool at runtime.

To make a file available to a tool account:

# Copy the file into the tool account directory.
# Become the tool account.
# Transfer ownership of the file to the tool account using the take command.

For example:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
yourshellaccountname@tools-sgebastion-10:~$ cp somefile ~tools.toolaccount/

yourshellaccountname@tools-sgebastion-10:~$ become toolaccount
tools.toolaccount@tools-sgebastion-10:~$ take somefile
&lt;/syntaxhighlight&gt;Store shared files in locations that are appropriate for their purpose:

* Use &quot;/data/project/toolname&quot; for persistent tool data.
* Use &quot;/data/scratch&quot; for temporary data that can be recreated.

=== Share code using packages or version control ===
For source code and reusable libraries, use version control or package management systems instead of manually copying files between tools.

* Version control systems such as Git provide a canonical source for code, make changes easier to track, and simplify collaboration between maintainers.
* Package managers can be used to distribute reusable code across multiple tools without duplicating files. This helps reduce maintenance overhead and ensures that multiple tools can use the same version of shared code.

When possible:

* Store source code in a version control repository.
* Distribute reusable libraries as packages.
* Use shared storage for runtime data rather than source code.
* Avoid maintaining multiple copies of the same code across different tools.

== Common problems ==

=== Permission denied ===
Sometimes the file system permissions on a tool's project directory (&lt;code&gt;/data/project/$TOOL&lt;/code&gt;) can become misconfigured and lose group write permission. This can prevent tool maintainers from uploading, modifying, or deleting files.

To check the current permissions, log in to a bastion host and inspect the directory:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
$ ssh login.toolforge.org
$ ls -ld /data/project/bd808-test
drwxrwsr-x 7 tools.bd808-test tools.bd808-test 4096 Jul 19 15:07 /data/project/bd808-test/
&lt;/syntaxhighlight&gt;The &lt;code&gt;drwxrwsr-x&lt;/code&gt; section of the output describes the directory permissions:

* &lt;code&gt;d&lt;/code&gt; – a directory
* &lt;code&gt;rwx&lt;/code&gt; – permissions for the owning user:
** &lt;code&gt;r&lt;/code&gt; – read
** &lt;code&gt;w&lt;/code&gt; – write
** &lt;code&gt;x&lt;/code&gt; – execute
* &lt;code&gt;rws&lt;/code&gt; – permissions for the owning group:
** &lt;code&gt;r&lt;/code&gt; – read
** &lt;code&gt;w&lt;/code&gt; – write
** &lt;code&gt;s&lt;/code&gt; – sticky. Sticky implies execute and also attempts to set the same group ownership on all new files and directories created inside the directory.
* &lt;code&gt;r-x&lt;/code&gt; – permissions for other users:
** &lt;code&gt;r&lt;/code&gt; – read
** &lt;code&gt;x&lt;/code&gt; – execute
** &lt;code&gt;-&lt;/code&gt; – no write permission

A common cause of permission errors is that the group write permission (&lt;code&gt;w&lt;/code&gt;) is missing from the tool's directory or one of its files or subdirectories.

=== Ownership issues ===
Members of a tool account belong to the tool's owning group. When the correct group permissions are missing, tool maintainers may be unable to upload, modify, or manage files.

To restore group write permissions, log in to a bastion host, become the tool account, and run:&lt;syntaxhighlight lang=&quot;bash&quot;&gt;
$ ssh login.toolforge.org
$ become $MY_TOOL
$ chmod -R g+w /data/project/$MY_TOOL
&lt;/syntaxhighlight&gt;

This command recursively grants write permission to the tool's group for all files and directories within the tool's project directory.

=== Transfer problems ===
Incorrect ownership or file permissions can also prevent files from being transferred or updated successfully.

If transferred files cannot be modified by a tool account, check that:

* The files were copied to the correct location.
* The tool account belongs to the owning group.
* The destination directory has group write permission.
* Files and subdirectories have inherited the correct permissions.

If necessary, restore the appropriate group write permissions as described in [[Help:Toolforge/Managing and sharing files#Ownership issues|Ownership issues]].

== Best practices ==
Follow these recommendations when managing and sharing files in Toolforge:

* Store sensitive files, such as passwords, API keys, and configuration files, with restricted permissions.
* Use the least permissive file permissions that allow your tool to function correctly.
* Choose the appropriate storage location for your files based on whether they need to be private, shared between tools, or publicly accessible.
* Use tool accounts and shared storage to collaborate instead of duplicating files across multiple tools.
* Prefer package managers or version control systems, such as Git, for sharing reusable code between projects.
* Verify file ownership and permissions after transferring files to avoid access issues.
* Use &lt;code&gt;rsync&lt;/code&gt; for large or repeated file transfers and synchronization tasks.
* Regularly review file permissions, especially after moving or modifying files.

== Related documentation ==
For more information about managing files and storage in Toolforge, see:

* [[Help:Toolforge/Tool accounts]]
* [[Help:Shared storage]]
* [[Help:Access to Toolforge instances with PuTTY and WinSCP]]
* [[Help:Toolforge/Developing successful tools]]
* [[Help:Toolforge]]</text>
      <sha1>af3jw8sl7xdeszkr8et516i0trsykwt</sha1>
    </revision>
  </page>
  <page>
    <title>Blazegraph Migration: Rewrite of GAS</title>
    <ns>0</ns>
    <id>460366</id>
    <revision>
      <id>2433184</id>
      <parentid>2429941</parentid>
      <timestamp>2026-07-05T02:41:53Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>lang=&quot;sparql&quot;; wikify</comment>
      <origin>2433184</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="31845" sha1="fv3i0z522u3phkzoouk7l63vinoma96" xml:space="preserve">The [[github:blazegraph/database/wiki/RDF_GAS_API|Blazegraph GAS]] (Gather-Apply-Scatter) service provides a server-side, graph traversal abstraction for Wikidata. It is exposed through a custom SPARQL service and implements the Java IGASProgram classes - BFS (breadth-first search), SSSP (single source shortest path), CC (connected components), and PageRank (PR) features. However, '''given how Wikidata’s RDF is constructed, BFS and SSSP are the only traversal mechanisms available''' (reasons are explained in detail in the sections below), and the only ones considered for rewrite.

When porting this functionality to SPARQL 1.1, many factors must be considered. For example, the closest SPARQL-native primitive for BFS traversal is to use property paths. They support forward/reverse/alternative/sequence paths and arbitrary-length (*, +, ?) connectivity. However, taking a property path approach does not achieve parity with GAS since intermediate vertices, predecessor nodes, path length, and path count are not exposed. Another issue with unbounded property paths (* and +) is the likelihood of timeouts.

When using the public Wikidata query service, solutions must fit into the 60-second timeout deadline. Therefore, they must be bounded and selective. For example, a GAS algorithm could be used to retrieve graph nodes starting from a source and expanding out based on the parameters given. To ensure completion, the request should always be explicitly reformulated as ''“retrieve up to depth k, over this fixed property set, from these seed nodes, returning at most this much data.”'' 

Details are provided below regarding rewriting GAS service queries using only SPARQL 1.1. A vendor-specific alternative is also available using the [https://docs.qlever.dev/path-search/ QLever-specific pathSearch service]. Migrating to a SPARQL 1.1 rewrite is future-proof. However, for transparency, it is acknowledged that QLever's pathSearch rewrite is more performant and less cumbersome. 

Although pathSearch is yet another vendor-specific extension, the service has been in place since July 2025 and addresses a problem space not adequately solved by SPARQL alone. QLever's pathSearch is presented in this document as the recommended alternative for the GAS service. However, there is no guarantee that a similar solution will be available in other (future) triple stores, and WDQS users will be dependent on QLever to maintain the service. Also note that using the pathSearch service still requires additional query clauses to achieve equivalency with GAS. This is discussed in more detail below.

== Rewrite Patterns Overview ==
Various approaches are described for rewriting GAS BFS (breadth-first search) queries. In addition, there are three other gas:service classes - SSSP (single source shortest path), CC (connected-components) and PageRank (PR). These are either functionally equivalent to BFS or not possible to execute on Blazegraph for Wikidata. The specific details are:

* SSSP - The Blazegraph implementation defines edge “weights” by reading link attributes from RDR-encoded statements (reification-done-right)&lt;ref&gt;Reification-Done-Right uses Blazegraph’s “quoted-triple” syntax - &lt;&lt;?s ?p ?o&gt;&gt; - to reference a triple statement and then define properties of that statement (such as starting date). For example, &lt;&lt;wd:Q43274 wdt:P26 wd:Q9685&gt;&gt; pq:P580 &quot;1981-07-29T00:00:00Z&quot; can be translated as &quot;the statement that Charles married Diana has a start date of 29 July 1981&quot;. Alternatively, Wikidata only uses standard reification practices to define a statement and then the properties of that statement - e.g., wd:Q43274 p:P26 stmt_IRI . stmt_IRI ps:P26 wd:Q9685 . stmt_IRI pq:P580 “1981-07-29T00:00:00Z”.&lt;/ref&gt;. But, there are no RDR statements in Wikidata. Every edge is a normal subject-predicate-object triple. Therefore, SSSP falls back to unit weights on every edge and becomes BFS.
* CC -  The algorithm's design assumes the graph is small and begins by initializing its frontier with all vertices. On a Wikidata-scale store, the initialization pass alone exceeds query timeouts. Unfortunately, the use of the gas:in parameter does nothing to scope the algorithm. 
* PR - The implementation begins as does CC, with an initialization of all vertices. Similarly, it times out on Wikidata.

For this analysis, it is assumed that a gas:linkType service parameter is always defined. If omitted, then GAS examines every predicate/triple in Wikidata - which would cause a Blazegraph timeout. Because of this, it is assumed that a path’s property/link is always specified.

Also, note that three of the GAS parameters are not discussed in this document as they have no relevance in a Wikidata context. They are: 

* gas:nthreads, which is a Blazegraph-specific server parallelism hint
* gas:linkAttrType, which selects and filters traversal to edges with a specified edge property type, and appears to be tied strictly to Blazegraph’s reification-done-right (RDR) statement declaration  
* gas:scheduler, which is not exposed in WDQS

== Breadth First Search ==
Blazegraph's GAS BFS is a scatter-only breadth-first traversal beginning at one or more seed vertices (defined via gas:in), following edges (matching gas:linkType) in the direction set by the gas:traversalDirection parameter (Forward, Reverse, or Undirected). Results are reported regarding each visited vertex - hop-count depth output as the variable in gas:out1, and discovery predecessor from gas:out2. 

Each vertex is visited exactly once. The execution thread to encounter it “wins”, setting both depth and predecessor. Subsequent visitors contribute nothing to the results, which enforces shortest-path semantics for predecessors. Bounds are enforced via gas:maxIterations (which is a hard depth cap), gas:maxVisited (which caps the number of vertices visited), and gas:target with gas:maxIterationsAfterTargets. The latter, maxIterationsAfterTargets, is the only mechanism that gives early termination but only when specific target endpoints are reached&lt;ref&gt; If gas:target is supplied without gas:maxIterationsAfterTargets, the algorithm runs until termination and then prunes the results, walking backwards from each target to a seed, retaining only vertices along those paths. (This acts as a post-processing filter.) With maxIterationsAfterTargets, the algorithm actively terminates if the target is encountered. For BFS, a value &gt; 0 has no value.&lt;/ref&gt;. 

The following table presents the GAS service parameters:
{| class=&quot;wikitable&quot;
|-
! Parameter !! Type !! Effect
|-
| gas:in || Multi-valued IRI/QID || Initial frontier
|-
| gas:linkType || Single-valued PID || Edge predicate defining the path
|-
| gas:traversalDirection || Enum: Forward, Reverse, Undirected || Edge direction
|-
| gas:maxIterations || Integer || Number of iterations/depth
|-
| gas:Target || Multi-valued IRI/QID || Output filter
|-
| gas:maxIterationsAfterTargets || Integer || Early termination if a target is defined and found
|-
| gas:maxVisited || Integer || Number of vertices visited 
|-
| gas:out || Variable || IRI of a visited vertex
|-
| gas:out1 || Variable || Depth of the visited vertex
|-
| gas:out2 || Variable || Predecessor IRI for a visited vertex
|}

A GAS service query takes the form:
&lt;syntaxhighlight lang=&quot;text&quot;&gt;
SERVICE gas:service {
    gas:program gas:gasClass &quot;com.bigdata.rdf.graph.analytics.BFS&quot; ;
                gas:in &lt;seed_iri&gt; ;              # May be more than 1
                gas:linkType &lt;predicate_iri&gt; ;   # May be more than 1
                gas:traversalDirection “Forward” ;    # Default
                gas:maxIterations 5 ;            # Always valuable to specify a limit
                gas:target &lt;target_iri&gt; ;        # Optional, may be more than 1
                gas:maxIterationsAfterTargets 0 ;   # Only relevant with gas:target
                gas:maxVisited 1000 ;            # Optional
                gas:out      ?vertex ;
                gas:out1     ?depth ;
                gas:out2     ?predecessor .
}
&lt;/syntaxhighlight&gt;

The following sections discuss the different aspects related to rewriting a GAS query - either as SPARQL 1.1 or using QLever’s pathSearch.

In many respects, rewriting with pathSearch is more straightforward since there is a logical correspondence between the pathSearch parameters and GAS’. This is shown in the table directly below:
{| class=&quot;wikitable&quot;
|-
! GAS !! QLever Analog !! Comments
|-
| gas:in &lt;QID&gt; || pathSearch:source &lt;QID&gt; || Required for both
|-
| gas:out2 ?predecessor || pathSource:start ?predecessor || Required for QLever; Note that for GAS, at depth 0, ?predecessor = “:in” QID 
|-
| gas:target &lt;QID&gt; || pathSearch:target &lt;QID&gt; || Optional for both; Can be multi-valued but has processing implications for QLever&lt;ref&gt;If multiple start AND target QIDs are defined, these should be paired and pathSearch:cartesian false added as a parameter to limit an explosion in the number of paths.&lt;/ref&gt;
|-
| gas:out ?vertex || pathSearch:end ?vertex || Required for both
|-
| gas:linkType &lt;PID&gt; || Nested subquery binding - ?predecessor &lt;PID&gt; ?vertex || Required, single-valued for GAS
|-
| gas:out1 ?depth || pathSearch:edgeColumn ?edge || depth&lt;ref&gt;edge != depth because (a) an edge is a hop between two node-depths, off by one from the starting node, and (b) edgeColumn defines the position in each path. To recover BFS-style depth, two corrections are needed: take the end node's edge + 1, then MIN(...) over all paths to the node.&lt;/ref&gt; = QLever’s ?edge + 1
|-
| gas:maxIterations # || pathSearch:maxDepth # || 5 or less&lt;ref&gt;The initial gate depth is 5. This value may be changed in the future, and/or larger depth values allowed for authorized users.&lt;/ref&gt;
|-
|   || pathSearch:pathColumn ?p || Must be specified in the query but is unused
|}

It is important to note that there is a fundamental difference in the algorithms. Whereas GAS uses a breadth-first search approach (limiting its path traversal by the first encounter of a node), QLever’s pathSearch takes a depth-first approach and is only limited by its “target” and/or maxDepth parameter value.

Another difference is in the result sets. For GAS, the result is the set of visited vertices with shortest-path metadata - one row per vertex, ties silently broken, non-shortest paths invisible. For pathSearch (which walks each path to maxDepth before backtracking), the result enumerates each path. A  vertex reachable by N different paths appears in N rows. The result is therefore a bag of paths, including non-shortest ones up to whatever maxDepth bound is set. This would also be the result of a SPARQL 1.1-compliant query.

Although that last paragraph implied that GAS equivalence is not possible, that is not true. The following processing pattern can be applied to both SPARQL and pathSearch:
# Get the paths up to N depth
# For each vertex, find MIN(depth) across all paths reaching it
# Filter to keep only paths at that minimum depth (or equivalently: filter at MIN(depth) per vertex)
# SAMPLE (or otherwise pick one) predecessor from the surviving paths
# Return the vertex, depth, predecessor based on the SAMPLEd vertex 

Explicit examples are provided in the remainder of this document.

== Reachability and Depth ==
Starting simple, this section examines a request to find the children/grandchildren/great-grandchildren/… (P40) of a person (Q9682, Queen Elizabeth). The Blazegraph query is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?depth WHERE {
    SERVICE gas:service {
       gas:program gas:gasClass &quot;com.bigdata.rdf.graph.analytics.BFS&quot; ;
                  gas:in wd:Q9682 ;            
                  gas:linkType wdt:P40 ;   
                  gas:maxIterations 5 ; 
                  gas:out ?vertex ;                   
                  gas:out1 ?depth }
 } ORDER BY ?depth
&lt;/syntaxhighlight&gt;

A subset of the results (27 results in total) is: 

::[[File:Rewrite for Blazegraph GAS Service - Fig 1.png|Figure 1]]

We could modify this to replace the QIDs by their labels, but that is an exercise left to the reader.

In total, there are 3 levels of child, grandchild and great-grandchild from wd:Q9682 (Elizabeth II), but the limit (maxIterations) defined in the query was 5. The GAS algorithm simply ran out of nodes to explore using the P40 predicate.

An equivalent SPARQL 1.1 query is ''not feasible if there is no limit on depth''. Why? 
* Either, the query will be so deep as to exceed the 60 second timeout
* Or, if explicitly rewritten (wdt:P40/wdt:P40/wdt:P40/…/wdt:P40, the author will tire of defining the depth

For a simple query such as above, known to have a limited path, a query rewrite can be accomplished using a property path approach. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT DISTINCT ?vertex WHERE {
    wd:Q9682 wdt:P40* ?vertex 
}
&lt;/syntaxhighlight&gt;

But, this fails because it is not possible to get the depth variable. So, if all that you are interested in are the vertices - then this is sufficient. 

Also, it is valuable to note that the property path “*” includes zero-length paths. So, it includes the initial gas:in value (which GAS reports as depth 0). If the property path is changed to “+”, then the subject node (in this case, wd:Q9682) is not reported. 

To achieve equivalent results with MINIMUM depth and to return the gas:in value, the query must be explicitly written as: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex (MIN(?d) AS ?depth) WHERE {
    { { BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d) }
      UNION { wd:Q9682 wdt:P40 ?vertex . BIND(1 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40 ?vertex . BIND(2 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(3 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(4 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(5 AS ?d) } }
} GROUP BY ?vertex ORDER BY ?depth 
&lt;/syntaxhighlight&gt;

On QLever, running this query, the same 27 results were returned.

Performing the same rewrite using QLever’s pathSearch, the query becomes: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
PREFIX pathSearch: &lt;https://qlever.cs.uni-freiburg.de/pathSearch/&gt;
SELECT ?vertex (MIN(?d) AS ?depth) WHERE {
  { { BIND(wd:Q9682 AS ?vertex) . BIND (0 AS ?d) }     # Binding at depth 0 is still needed
    UNION
    { SERVICE pathSearch: {
        _:p pathSearch:algorithm  pathSearch:allPaths ;
            pathSearch:source wd:Q9682 ;
            pathSearch:maxDepth 5; 
            pathSearch:start ?predecessor ;  
            pathSearch:end ?vertex ;
            pathSearch:pathColumn ?pc ;
            pathSearch:edgeColumn ?edge .
        { SELECT * WHERE { ?predecessor wdt:P40 ?vertex . } } }   # Defines “path”
        BIND(?edge + 1 AS ?d) } }
} GROUP BY ?vertex ORDER BY ?depth
&lt;/syntaxhighlight&gt;

This query ran in shorter time and returned the same 27 results.

Certainly, things can be simplified if depth=0 binding is not needed, edge = depth -1 is acceptable to return, or all possible paths (versus shortest path) are sufficient.

Note the traversed edge must be defined for the path. In GAS, this is accomplished by specifying the gas:linkType. In QLever, it is accomplished by specifying the predicate of a subquery nested in the SERVICE request&lt;ref&gt;Although the subquery can be more complex than a single triple pattern, only a single triple is needed for equivalency with Blazegraph’s single gas:linkType parameter.&lt;/ref&gt;. 

== Traversal Direction ==
To affect GAS’ traversal direction, the following is defined as a parameter:
* gas:traversalDirection &quot;Forward&quot;
* gas:traversalDirection &quot;Reverse&quot;
* gas:traversalDirection &quot;Undirected&quot;

To achieve the equivalent results using SPARQL 1.1, the property path expression is changed. The corresponding paths are:

* Forward: start_QID wdt:P40* ?vertex (or start_QID wdt:P40/…/wdt:P40 ?vertex)
* Reverse: start_QID ^wdt:P40* ?vertex (or start_QID ^wdt:P40/…/^wdt:P40 ?vertex, or handled by reversing the subject and object&lt;ref&gt;Reversing the subject and object may improve performance but can cause semantic complications - as discussed next. This needs to be taken into consideration.&lt;/ref&gt; - ?vertex wdt:P40* start_QID)
* Undirected: start_QID (wdt:P40|^wdt:P40)* ?vertex (or start_QID (wdt:P40|^wdt:P40)/…/(wdt:P40|^wdt:P40) ?vertex)

As noted above, the property path “*” always includes the source node.

To achieve the results using QLever’s pathSearch, the nested subquery is changed. The corresponding subqueries are:
* Forward: { SELECT * WHERE { ?predecessor wdt:P40 ?vertex . } }
* Reverse: { SELECT * WHERE { ?vertex wdt:P40 ?predecessor . } }
* Undirected: { SELECT * WHERE { { ?predecessor wdt:P40 ?vertex } UNION { ?vertex wdt:P40 ?predecessor } } }

== Predecessor Approximation (Increasing Complexity) ==
In BFS, a “predecessor” is the ''parent'' vertex that causes another node to be included in the traversal path (e.g., the node through which the new vertex was first encountered). 

Using the gas:service, a request for predecessors is defined by adding gas:out2 ?predecessor ; to the query from above, as well as adding ?predecessor to the SELECT variables.

The results for this modified query are shown below (still 27 results):
:[[File:Rewrite for Blazegraph GAS Service - Fig 2.png|Figure 2]]

But … it is not possible to obtain equivalent results using SPARQL 1.1 in all cases. We can, however, obtain a close “approximation”. Approximations are needed when the property path traversal can encounter a node several times - at different “depths”, with different predecessors. As noted, GAS returns the ''first'' encounter&lt;ref&gt;However, with a change to the underlying RDF triples, the first encounter also has the possibility to change.&lt;/ref&gt;. 

To create the most precise “approximation”, the equivalent SPARQL 1.1 query must be defined with an embedded SELECT statement. The latter returns the first occurrence of a node (which is accomplished with the MIN(...) binding). Then, we must guarantee that we return a single predecessor (because again, there could be duplicates) by using SAMPLE. The situation is further complicated because we need to return a single predecessor from the minimum depth. This is accomplished by adding a FILTER constraint to force the depth at which the predecessor is selected to be the same as the minimum depth. 

Restating the above, Blazegraph BFS picks one predecessor per visited vertex (the one that first caused the visit). But, the SPARQL SAMPLE predecessor returns one of any of “the predecessors at minimum depth&quot;. This is valid as first-discovery semantics, but may differ from Blazegraph's choice when multiple parents tie at the minimum depth. 

Taking all this into consideration, the final SPARQL query is written as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?depth (SAMPLE(?pred) AS ?predecessor) WHERE {
    { SELECT ?vertex (MIN(?d1) AS ?depth) WHERE {
        {{ BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d1) }
         UNION { wd:Q9682 wdt:P40 ?vertex . BIND(1 AS ?d1) }
         UNION { wd:Q9682 wdt:P40/wdt:P40 ?vertex . BIND(2 AS ?d1) }
         UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(3 AS ?d1) }}
      } GROUP BY ?vertex          # Get minimum depth per vertex
    }
    {{ BIND(wd:Q9682 AS ?vertex). BIND (0 AS ?d2). }
     UNION { wd:Q9682 wdt:P40 ?vertex . BIND(1 AS ?d2) . BIND(wd:Q9682 AS ?pred) }
     UNION { wd:Q9682 wdt:P40 ?v1 . ?v1 wdt:P40 ?vertex . 
                    BIND(2 AS ?d2) . BIND(?v1 AS ?pred) }
     UNION { wd:Q9682 wdt:P40/wdt:P40 ?v2 . ?v2 wdt:P40 ?vertex .
                    BIND(3 AS ?d2) . BIND(?v2 AS ?pred) }
    }      # Get all paths 3 levels deep
    FILTER(?d2 = ?depth)      # Filter paths that are not the minimum depth
  } GROUP BY ?vertex ?depth ORDER BY ?depth
&lt;/syntaxhighlight&gt;

Which returns the same 27 values in the same order.

Given the complexity of this discussion, it is worthwhile to include rewrite details when the traversal direction is “Reverse”. When the GAS query is reversed, it is especially valuable to include a maxIterations limit (let’s say 3 for ease of rewriting the queries) to bound the results. This makes sense when considering the intent of the query. In the forward direction, the request is for the children of Queen Elizabeth II (a small, bounded group). But, in the reverse direction, it asks about Elizabeth II’s complete ancestry path (not a small group!). 

When executed, there are 15 results up to depth 3.

When rewritten, the query can take one of two forms depending on whether the inverse property path “^” is used, or the order of the subject/object values of the triples are reversed. 

For the first case (using “^”), the query is the same, but wdt:P40 is replaced by ^wdt:P40. Also, the same 15 results are obtained.

In the second case (reversing the subject/object variables), the query becomes:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?depth (SAMPLE(?pred) AS ?predecessor) WHERE {
 { SELECT ?vertex (MIN(?d1) AS ?depth) WHERE {
     {  { BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d1) }
      UNION { ?vertex wdt:P40 wd:Q9682 . BIND(1 AS ?d1) }
      UNION { ?vertex wdt:P40/wdt:P40 wd:Q9682 . BIND(2 AS ?d1) }
      UNION { ?vertex wdt:P40/wdt:P40/wdt:P40 wd:Q9682 . BIND(3 AS ?d1) } }
   } GROUP BY ?vertex
 }
 {  { BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d2) }
  UNION { ?vertex wdt:P40 wd:Q9682 .
          BIND(1 AS ?d2) . BIND(wd:Q9682 AS ?pred) }
  UNION { ?vertex wdt:P40 ?v1 . ?v1 wdt:P40 wd:Q9682 .
          BIND(2 AS ?d2) . BIND(?v1 AS ?pred) }
  UNION { ?vertex wdt:P40 ?v1 . ?v1 wdt:P40 ?v2 . ?v2 wdt:P40 wd:Q9682 .
          BIND(3 AS ?d2) . BIND(?v1 AS ?pred) }
 }
 FILTER(?d2 = ?depth)
} GROUP BY ?vertex ?depth ORDER BY ?depth
&lt;/syntaxhighlight&gt;

If this also seems equivalent, look more closely. The ''variable ?v1 is bound as the predecessor at all depths of 2 or more''! This seems to make no sense until you consider the semantics. At depth 3:
* With ^: The predecessor is ?v2, where the chain is Q9682 → v1 → v2 → vertex and BFS depths are 0, 1, 2, 3.
* Without ^: The chain is vertex → v1 → v2 → Q9682 and BFS depths run 3, 2, 1, 0. The predecessor of the ?vertex node is always ?v1 (and this is true for all depths &gt;= 3).

It is valuable to note that (typically) the second syntax (reversing the subject/object) is a slightly more performant query than using the inverse property path (“^”). However, remembering to consider the direction of traversal is a major headache for the query author.  

When performing the rewrite using QLever’s pathSearch, there is a similar need for MIN() and SAMPLE() since a node can have several distinct parents among its paths.

Using pathSearch, the “forward” direction query is structurally similar to the SPARQL 1.1 definition:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
PREFIX pathSearch: &lt;https://qlever.cs.uni-freiburg.de/pathSearch/&gt;
SELECT DISTINCT ?vertex ?depth (SAMPLE(?pred) AS ?predecessor) WHERE {
   { SELECT ?vertex (MIN(?d1) AS ?depth) WHERE {
       { SERVICE pathSearch: {
           _:p pathSearch:algorithm pathSearch:allPaths ;
                 pathSearch:source wd:Q9682 ;
                 pathSearch:maxDepth 3;  
                 pathSearch:start ?pred ; 
	    pathSearch:end ?vertex ;
                 pathSearch:pathColumn ?pc ; 
	    pathSearch:edgeColumn ?edge .
            { SELECT * WHERE { ?pred wdt:P40 ?vertex . } } }
         BIND(?edge + 1 AS ?d1) 
      }
      UNION { BIND(wd:Q9682 AS ?vertex) BIND(0 AS ?d1) }
    } GROUP BY ?vertex }
   { { SERVICE pathSearch: {
         _:p pathSearch:algorithm pathSearch:allPaths ;
               pathSearch:source wd:Q9682 ;
               pathSearch:maxDepth 3; 
               pathSearch:start ?pred ; 
	  pathSearch:end ?vertex ;
               pathSearch:pathColumn ?pc ; 
	  pathSearch:edgeColumn ?edge .
          { SELECT * WHERE { ?pred wdt:P40 ?vertex . } } }
       BIND(?edge + 1 AS ?d2) 
    }
    UNION { BIND(wd:Q9682 AS ?vertex) BIND(0 AS ?d2) } }  
  FILTER(?d2 = ?depth)
} GROUP BY ?vertex ?depth ORDER BY ?depth
&lt;/syntaxhighlight&gt;

There are four items to note: 
* Reporting the predecessor is “free” in pathSearch since it is the pathSearch:start variable
* It is not necessary to explicitly define intermediate nodes (Q9682 → ?v1 → ?v2 → …) because pathSearch carries each edge's source
* The starting/source node has no predecessor in the GAS service results. Therefore,  the depth-0 UNION branch only binds the variables, ?out and ?d2, but leaves the ?pred variable unbound; SAMPLE(?pred) yields null for Q9682
* Although the above is complex, both SERVICE pathSearch definitions are exactly the same, making rewrite straightforward

As an added benefit, there is no need to worry about the “reverse” query since it is constructed by switching the subject and object variables in the nested subquery (?vertex wdt:P40 ?pred).

== Considering GAS Target ==
An additional, possible parameter to the BFS query is “gas:target”. This addition returns only those paths that include the “target” node(s) - leading ''from'' the ‘target’ ''back to'' the “in”/starting node.

When rewriting the query in SPARQL 1.1, it is not possible to define a traversal path from a source to a specific target. This is because it is not possible to stop a query execution at a target. Searching for a target (or multiple targets) has to be performed after the query has completed and returned its results.

Filtering for a specific target relies entirely on the predecessor results. This behavior must be done on the client-side by following these steps:
# Execute the rewritten query specified above.
# Check whether the TARGET node appears in the results.
# If yes, start at the TARGET.
# Repeatedly follow the predecessor nodes until you reach the starting node. 
# Keep only those vertices.

For example, if you have a result that appears as:
::{| class=&quot;wikitable&quot;
|-
! ?vertex !! ?depth !! ?predecessor
|-
| A || 0 ||  
|-
| B || 1 || A
|-
| C || 1 || A
|-
| D || 2 || B
|-
| E || 2 || C
|-
| F || 3 || D
|-
| G || 4 || F
|}

If gas:target is F, the path to the target is:  A -&gt; B -&gt; D -&gt; F, and the reported nodes would be A, B, D, F. 

Working backwards from F in a SPARQL result set, starts at the ?vertex variable = F, ?predecessor = D. Then, ?vertex = D, ?predecessor = B. Then, ?vertex = B, ?predecessor = A (the starting node).

Note that multiple gas:targets can be defined. If this is requested, then the 5 steps above need to be repeated for each target. 

However, performing this on QLever is straightforward. The pathSearch:target parameter is used. It performs the same role as gas:target plus gas:maxIterationsAfterTarget, stopping the search when the target endpoint is reached (if the target is encountered within the specified maxIterations/maxDepth limit).

== Multiple Frontiers ==
GAS’ “frontier” is the set of nodes that participate in the next round of gather/apply/scatter steps. Basically, it is the algorithm’s working set. GAS’ ''initial'' frontier(s) are the node(s) that are defined using the service parameter, gas:in. Note that there may be more than one.  

Examining a sample Blazegraph query using the gas:service with multiple frontiers:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?out ?depth WHERE {
    SERVICE gas:service {
      gas:program gas:gasClass &quot;com.bigdata.rdf.graph.analytics.BFS&quot; ;
                  gas:in wd:Q9682 ;        # Where to start
                  gas:in wd:Q42 ;        
                  gas:linkType wdt:P40 ;   # What property to follow 
                  gas:out ?out ;           # The next vertex (FORWARD traversal)
                  gas:out1 ?depth }
 } ORDER BY ?depth
&lt;/syntaxhighlight&gt;

This query returns the following 29 results:
::[[File:Rewrite for Blazegraph GAS Service - Fig 3.png|Figure 3]]

Reviewing the table, one sees that it is useful for retrieving a list of all descendents, but does not provide any information on the family tree (the starting node) to which the descendent belongs. This is important when there are multiple starting nodes. Also, the query results do not indicate if a descendent occurs in both trees since GAS returns only the first encounter. 

Adding a ?predecessor variable helps a bit, but past the first generation still requires work to assemble the complete tree. 

Rewriting the query as follows provides an equivalent query and also indicates the initial/starting node: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?out (MIN(?d) AS ?depth) WHERE {
    VALUES ?vertex { wd:Q9682 wd:Q42 }             # Specifying the gas:in seeds
    {{ VALUES ?vertex { wd:Q9682 wd:Q42 } BIND(?vertex AS ?out) . BIND(0 AS ?d) }
     UNION { ?vertex wdt:P40 ?out . BIND(1 AS ?d) }
     UNION { ?vertex wdt:P40/wdt:P40 ?out . BIND(2 AS ?d) }
     UNION { ?vertex wdt:P40/wdt:P40/wdt:P40 ?out . BIND(3 AS ?d) }}
} GROUP BY ?vertex ?out ORDER BY ?depth
&lt;/syntaxhighlight&gt;

Note that it is still necessary to limit and explicitly define the query depth. 

It is important to note that the VALUES statement is repeated in the query since a variable within a UNION branch cannot see a binding/VALUES for it that is defined outside the UNION. The variable must be brought into the branch's scope.

A portion of the result set (29 members) is shown below: 
::[[File:Rewrite for Blazegraph GAS Service - Fig 4.png|Figure 4]]

Note that the query rewrite above gives additional results since it reports per-seed values. So, a descendant reachable from both of the seeds (Q42 and Q9682) would appear twice, once per seed. (Given that the seeds are Elizabeth II and Douglas Adams, this is very unlikely.) GAS’ results would merge the trees and report each descendant once with one of the predecessors. In this case, the rewrite intentionally produces a more-informative per-seed output. 

Similarly, this query can be written using QLever’s pathSearch. The rewrite is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
PREFIX pathSearch: &lt;https://qlever.cs.uni-freiburg.de/pathSearch/&gt;
SELECT ?vertex ?out (MIN(?d) AS ?depth) WHERE {
  { SERVICE pathSearch: {
       _:p pathSearch:algorithm  pathSearch:allPaths ;
             pathSearch:source wd:Q9682 ;       # Two sources
             pathSearch:source wd:Q42 ; 
             pathSearch:maxDepth 3; 
             pathSearch:start  ?vertex ;     
             pathSearch:end ?out ;
             pathSearch:pathColumn ?pc ; 
             pathSearch:edgeColumn ?edge .
       { SELECT * WHERE { ?vertex wdt:P40 ?out . } } }
    BIND(?edge + 1 AS ?d) }
  # Each source at depth 0
  UNION { VALUES ?vertex { wd:Q9682 wd:Q42 }
                 BIND(?vertex AS ?out) . BIND(0 AS ?d) }
} GROUP BY ?vertex ?out ORDER BY ?depth
&lt;/syntaxhighlight&gt;

It returns the same 29 results.

== Query Rewrite Patterns ==
Across the four GAS algorithms (BFS, SSSP, CC, PR) analyzed in this document, only one (BFS) results in an actual GAS-to-SPARQL rewrite. The other algorithms either reduce to BFS, or have no usable starting query that warrants rewriting. 

At this time, '''only BFS and SSSP queries will be rewritten and tested. Rewrites will utilize QLever’s pathSearch service.'''

The approach to rewriting will be based on the 1:1 correspondence of service parameters that was presented on page 5, along with always capturing the minimum depth as a sub-query, and doing a SAMPLEing of predecessors at the minimum depth in the output variables. 

The only other consideration is whether the gas:in/seed is returned. It is always returned unless the GAS query explicitly removes it by FILTERing for depth &gt; 0. If the latter is true, then the BINDing of the source QID as the pathSearch:start variable, at depth = 0, is unnecessary.

== Footnotes ==</text>
      <sha1>fv3i0z522u3phkzoouk7l63vinoma96</sha1>
    </revision>
    <revision>
      <id>2433185</id>
      <parentid>2433184</parentid>
      <timestamp>2026-07-05T02:42:39Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>lang=&quot;gas&quot;</comment>
      <origin>2433185</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="31844" sha1="2o5r62uhq6rpp0qi39pdqrji36b0kbr" xml:space="preserve">The [[github:blazegraph/database/wiki/RDF_GAS_API|Blazegraph GAS]] (Gather-Apply-Scatter) service provides a server-side, graph traversal abstraction for Wikidata. It is exposed through a custom SPARQL service and implements the Java IGASProgram classes - BFS (breadth-first search), SSSP (single source shortest path), CC (connected components), and PageRank (PR) features. However, '''given how Wikidata’s RDF is constructed, BFS and SSSP are the only traversal mechanisms available''' (reasons are explained in detail in the sections below), and the only ones considered for rewrite.

When porting this functionality to SPARQL 1.1, many factors must be considered. For example, the closest SPARQL-native primitive for BFS traversal is to use property paths. They support forward/reverse/alternative/sequence paths and arbitrary-length (*, +, ?) connectivity. However, taking a property path approach does not achieve parity with GAS since intermediate vertices, predecessor nodes, path length, and path count are not exposed. Another issue with unbounded property paths (* and +) is the likelihood of timeouts.

When using the public Wikidata query service, solutions must fit into the 60-second timeout deadline. Therefore, they must be bounded and selective. For example, a GAS algorithm could be used to retrieve graph nodes starting from a source and expanding out based on the parameters given. To ensure completion, the request should always be explicitly reformulated as ''“retrieve up to depth k, over this fixed property set, from these seed nodes, returning at most this much data.”'' 

Details are provided below regarding rewriting GAS service queries using only SPARQL 1.1. A vendor-specific alternative is also available using the [https://docs.qlever.dev/path-search/ QLever-specific pathSearch service]. Migrating to a SPARQL 1.1 rewrite is future-proof. However, for transparency, it is acknowledged that QLever's pathSearch rewrite is more performant and less cumbersome. 

Although pathSearch is yet another vendor-specific extension, the service has been in place since July 2025 and addresses a problem space not adequately solved by SPARQL alone. QLever's pathSearch is presented in this document as the recommended alternative for the GAS service. However, there is no guarantee that a similar solution will be available in other (future) triple stores, and WDQS users will be dependent on QLever to maintain the service. Also note that using the pathSearch service still requires additional query clauses to achieve equivalency with GAS. This is discussed in more detail below.

== Rewrite Patterns Overview ==
Various approaches are described for rewriting GAS BFS (breadth-first search) queries. In addition, there are three other gas:service classes - SSSP (single source shortest path), CC (connected-components) and PageRank (PR). These are either functionally equivalent to BFS or not possible to execute on Blazegraph for Wikidata. The specific details are:

* SSSP - The Blazegraph implementation defines edge “weights” by reading link attributes from RDR-encoded statements (reification-done-right)&lt;ref&gt;Reification-Done-Right uses Blazegraph’s “quoted-triple” syntax - &lt;&lt;?s ?p ?o&gt;&gt; - to reference a triple statement and then define properties of that statement (such as starting date). For example, &lt;&lt;wd:Q43274 wdt:P26 wd:Q9685&gt;&gt; pq:P580 &quot;1981-07-29T00:00:00Z&quot; can be translated as &quot;the statement that Charles married Diana has a start date of 29 July 1981&quot;. Alternatively, Wikidata only uses standard reification practices to define a statement and then the properties of that statement - e.g., wd:Q43274 p:P26 stmt_IRI . stmt_IRI ps:P26 wd:Q9685 . stmt_IRI pq:P580 “1981-07-29T00:00:00Z”.&lt;/ref&gt;. But, there are no RDR statements in Wikidata. Every edge is a normal subject-predicate-object triple. Therefore, SSSP falls back to unit weights on every edge and becomes BFS.
* CC -  The algorithm's design assumes the graph is small and begins by initializing its frontier with all vertices. On a Wikidata-scale store, the initialization pass alone exceeds query timeouts. Unfortunately, the use of the gas:in parameter does nothing to scope the algorithm. 
* PR - The implementation begins as does CC, with an initialization of all vertices. Similarly, it times out on Wikidata.

For this analysis, it is assumed that a gas:linkType service parameter is always defined. If omitted, then GAS examines every predicate/triple in Wikidata - which would cause a Blazegraph timeout. Because of this, it is assumed that a path’s property/link is always specified.

Also, note that three of the GAS parameters are not discussed in this document as they have no relevance in a Wikidata context. They are: 

* gas:nthreads, which is a Blazegraph-specific server parallelism hint
* gas:linkAttrType, which selects and filters traversal to edges with a specified edge property type, and appears to be tied strictly to Blazegraph’s reification-done-right (RDR) statement declaration  
* gas:scheduler, which is not exposed in WDQS

== Breadth First Search ==
Blazegraph's GAS BFS is a scatter-only breadth-first traversal beginning at one or more seed vertices (defined via gas:in), following edges (matching gas:linkType) in the direction set by the gas:traversalDirection parameter (Forward, Reverse, or Undirected). Results are reported regarding each visited vertex - hop-count depth output as the variable in gas:out1, and discovery predecessor from gas:out2. 

Each vertex is visited exactly once. The execution thread to encounter it “wins”, setting both depth and predecessor. Subsequent visitors contribute nothing to the results, which enforces shortest-path semantics for predecessors. Bounds are enforced via gas:maxIterations (which is a hard depth cap), gas:maxVisited (which caps the number of vertices visited), and gas:target with gas:maxIterationsAfterTargets. The latter, maxIterationsAfterTargets, is the only mechanism that gives early termination but only when specific target endpoints are reached&lt;ref&gt; If gas:target is supplied without gas:maxIterationsAfterTargets, the algorithm runs until termination and then prunes the results, walking backwards from each target to a seed, retaining only vertices along those paths. (This acts as a post-processing filter.) With maxIterationsAfterTargets, the algorithm actively terminates if the target is encountered. For BFS, a value &gt; 0 has no value.&lt;/ref&gt;. 

The following table presents the GAS service parameters:
{| class=&quot;wikitable&quot;
|-
! Parameter !! Type !! Effect
|-
| gas:in || Multi-valued IRI/QID || Initial frontier
|-
| gas:linkType || Single-valued PID || Edge predicate defining the path
|-
| gas:traversalDirection || Enum: Forward, Reverse, Undirected || Edge direction
|-
| gas:maxIterations || Integer || Number of iterations/depth
|-
| gas:Target || Multi-valued IRI/QID || Output filter
|-
| gas:maxIterationsAfterTargets || Integer || Early termination if a target is defined and found
|-
| gas:maxVisited || Integer || Number of vertices visited 
|-
| gas:out || Variable || IRI of a visited vertex
|-
| gas:out1 || Variable || Depth of the visited vertex
|-
| gas:out2 || Variable || Predecessor IRI for a visited vertex
|}

A GAS service query takes the form:
&lt;syntaxhighlight lang=&quot;gas&quot;&gt;
SERVICE gas:service {
    gas:program gas:gasClass &quot;com.bigdata.rdf.graph.analytics.BFS&quot; ;
                gas:in &lt;seed_iri&gt; ;              # May be more than 1
                gas:linkType &lt;predicate_iri&gt; ;   # May be more than 1
                gas:traversalDirection “Forward” ;    # Default
                gas:maxIterations 5 ;            # Always valuable to specify a limit
                gas:target &lt;target_iri&gt; ;        # Optional, may be more than 1
                gas:maxIterationsAfterTargets 0 ;   # Only relevant with gas:target
                gas:maxVisited 1000 ;            # Optional
                gas:out      ?vertex ;
                gas:out1     ?depth ;
                gas:out2     ?predecessor .
}
&lt;/syntaxhighlight&gt;

The following sections discuss the different aspects related to rewriting a GAS query - either as SPARQL 1.1 or using QLever’s pathSearch.

In many respects, rewriting with pathSearch is more straightforward since there is a logical correspondence between the pathSearch parameters and GAS’. This is shown in the table directly below:
{| class=&quot;wikitable&quot;
|-
! GAS !! QLever Analog !! Comments
|-
| gas:in &lt;QID&gt; || pathSearch:source &lt;QID&gt; || Required for both
|-
| gas:out2 ?predecessor || pathSource:start ?predecessor || Required for QLever; Note that for GAS, at depth 0, ?predecessor = “:in” QID 
|-
| gas:target &lt;QID&gt; || pathSearch:target &lt;QID&gt; || Optional for both; Can be multi-valued but has processing implications for QLever&lt;ref&gt;If multiple start AND target QIDs are defined, these should be paired and pathSearch:cartesian false added as a parameter to limit an explosion in the number of paths.&lt;/ref&gt;
|-
| gas:out ?vertex || pathSearch:end ?vertex || Required for both
|-
| gas:linkType &lt;PID&gt; || Nested subquery binding - ?predecessor &lt;PID&gt; ?vertex || Required, single-valued for GAS
|-
| gas:out1 ?depth || pathSearch:edgeColumn ?edge || depth&lt;ref&gt;edge != depth because (a) an edge is a hop between two node-depths, off by one from the starting node, and (b) edgeColumn defines the position in each path. To recover BFS-style depth, two corrections are needed: take the end node's edge + 1, then MIN(...) over all paths to the node.&lt;/ref&gt; = QLever’s ?edge + 1
|-
| gas:maxIterations # || pathSearch:maxDepth # || 5 or less&lt;ref&gt;The initial gate depth is 5. This value may be changed in the future, and/or larger depth values allowed for authorized users.&lt;/ref&gt;
|-
|   || pathSearch:pathColumn ?p || Must be specified in the query but is unused
|}

It is important to note that there is a fundamental difference in the algorithms. Whereas GAS uses a breadth-first search approach (limiting its path traversal by the first encounter of a node), QLever’s pathSearch takes a depth-first approach and is only limited by its “target” and/or maxDepth parameter value.

Another difference is in the result sets. For GAS, the result is the set of visited vertices with shortest-path metadata - one row per vertex, ties silently broken, non-shortest paths invisible. For pathSearch (which walks each path to maxDepth before backtracking), the result enumerates each path. A  vertex reachable by N different paths appears in N rows. The result is therefore a bag of paths, including non-shortest ones up to whatever maxDepth bound is set. This would also be the result of a SPARQL 1.1-compliant query.

Although that last paragraph implied that GAS equivalence is not possible, that is not true. The following processing pattern can be applied to both SPARQL and pathSearch:
# Get the paths up to N depth
# For each vertex, find MIN(depth) across all paths reaching it
# Filter to keep only paths at that minimum depth (or equivalently: filter at MIN(depth) per vertex)
# SAMPLE (or otherwise pick one) predecessor from the surviving paths
# Return the vertex, depth, predecessor based on the SAMPLEd vertex 

Explicit examples are provided in the remainder of this document.

== Reachability and Depth ==
Starting simple, this section examines a request to find the children/grandchildren/great-grandchildren/… (P40) of a person (Q9682, Queen Elizabeth). The Blazegraph query is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?depth WHERE {
    SERVICE gas:service {
       gas:program gas:gasClass &quot;com.bigdata.rdf.graph.analytics.BFS&quot; ;
                  gas:in wd:Q9682 ;            
                  gas:linkType wdt:P40 ;   
                  gas:maxIterations 5 ; 
                  gas:out ?vertex ;                   
                  gas:out1 ?depth }
 } ORDER BY ?depth
&lt;/syntaxhighlight&gt;

A subset of the results (27 results in total) is: 

::[[File:Rewrite for Blazegraph GAS Service - Fig 1.png|Figure 1]]

We could modify this to replace the QIDs by their labels, but that is an exercise left to the reader.

In total, there are 3 levels of child, grandchild and great-grandchild from wd:Q9682 (Elizabeth II), but the limit (maxIterations) defined in the query was 5. The GAS algorithm simply ran out of nodes to explore using the P40 predicate.

An equivalent SPARQL 1.1 query is ''not feasible if there is no limit on depth''. Why? 
* Either, the query will be so deep as to exceed the 60 second timeout
* Or, if explicitly rewritten (wdt:P40/wdt:P40/wdt:P40/…/wdt:P40, the author will tire of defining the depth

For a simple query such as above, known to have a limited path, a query rewrite can be accomplished using a property path approach. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT DISTINCT ?vertex WHERE {
    wd:Q9682 wdt:P40* ?vertex 
}
&lt;/syntaxhighlight&gt;

But, this fails because it is not possible to get the depth variable. So, if all that you are interested in are the vertices - then this is sufficient. 

Also, it is valuable to note that the property path “*” includes zero-length paths. So, it includes the initial gas:in value (which GAS reports as depth 0). If the property path is changed to “+”, then the subject node (in this case, wd:Q9682) is not reported. 

To achieve equivalent results with MINIMUM depth and to return the gas:in value, the query must be explicitly written as: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex (MIN(?d) AS ?depth) WHERE {
    { { BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d) }
      UNION { wd:Q9682 wdt:P40 ?vertex . BIND(1 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40 ?vertex . BIND(2 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(3 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(4 AS ?d) }
      UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(5 AS ?d) } }
} GROUP BY ?vertex ORDER BY ?depth 
&lt;/syntaxhighlight&gt;

On QLever, running this query, the same 27 results were returned.

Performing the same rewrite using QLever’s pathSearch, the query becomes: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
PREFIX pathSearch: &lt;https://qlever.cs.uni-freiburg.de/pathSearch/&gt;
SELECT ?vertex (MIN(?d) AS ?depth) WHERE {
  { { BIND(wd:Q9682 AS ?vertex) . BIND (0 AS ?d) }     # Binding at depth 0 is still needed
    UNION
    { SERVICE pathSearch: {
        _:p pathSearch:algorithm  pathSearch:allPaths ;
            pathSearch:source wd:Q9682 ;
            pathSearch:maxDepth 5; 
            pathSearch:start ?predecessor ;  
            pathSearch:end ?vertex ;
            pathSearch:pathColumn ?pc ;
            pathSearch:edgeColumn ?edge .
        { SELECT * WHERE { ?predecessor wdt:P40 ?vertex . } } }   # Defines “path”
        BIND(?edge + 1 AS ?d) } }
} GROUP BY ?vertex ORDER BY ?depth
&lt;/syntaxhighlight&gt;

This query ran in shorter time and returned the same 27 results.

Certainly, things can be simplified if depth=0 binding is not needed, edge = depth -1 is acceptable to return, or all possible paths (versus shortest path) are sufficient.

Note the traversed edge must be defined for the path. In GAS, this is accomplished by specifying the gas:linkType. In QLever, it is accomplished by specifying the predicate of a subquery nested in the SERVICE request&lt;ref&gt;Although the subquery can be more complex than a single triple pattern, only a single triple is needed for equivalency with Blazegraph’s single gas:linkType parameter.&lt;/ref&gt;. 

== Traversal Direction ==
To affect GAS’ traversal direction, the following is defined as a parameter:
* gas:traversalDirection &quot;Forward&quot;
* gas:traversalDirection &quot;Reverse&quot;
* gas:traversalDirection &quot;Undirected&quot;

To achieve the equivalent results using SPARQL 1.1, the property path expression is changed. The corresponding paths are:

* Forward: start_QID wdt:P40* ?vertex (or start_QID wdt:P40/…/wdt:P40 ?vertex)
* Reverse: start_QID ^wdt:P40* ?vertex (or start_QID ^wdt:P40/…/^wdt:P40 ?vertex, or handled by reversing the subject and object&lt;ref&gt;Reversing the subject and object may improve performance but can cause semantic complications - as discussed next. This needs to be taken into consideration.&lt;/ref&gt; - ?vertex wdt:P40* start_QID)
* Undirected: start_QID (wdt:P40|^wdt:P40)* ?vertex (or start_QID (wdt:P40|^wdt:P40)/…/(wdt:P40|^wdt:P40) ?vertex)

As noted above, the property path “*” always includes the source node.

To achieve the results using QLever’s pathSearch, the nested subquery is changed. The corresponding subqueries are:
* Forward: { SELECT * WHERE { ?predecessor wdt:P40 ?vertex . } }
* Reverse: { SELECT * WHERE { ?vertex wdt:P40 ?predecessor . } }
* Undirected: { SELECT * WHERE { { ?predecessor wdt:P40 ?vertex } UNION { ?vertex wdt:P40 ?predecessor } } }

== Predecessor Approximation (Increasing Complexity) ==
In BFS, a “predecessor” is the ''parent'' vertex that causes another node to be included in the traversal path (e.g., the node through which the new vertex was first encountered). 

Using the gas:service, a request for predecessors is defined by adding gas:out2 ?predecessor ; to the query from above, as well as adding ?predecessor to the SELECT variables.

The results for this modified query are shown below (still 27 results):
:[[File:Rewrite for Blazegraph GAS Service - Fig 2.png|Figure 2]]

But … it is not possible to obtain equivalent results using SPARQL 1.1 in all cases. We can, however, obtain a close “approximation”. Approximations are needed when the property path traversal can encounter a node several times - at different “depths”, with different predecessors. As noted, GAS returns the ''first'' encounter&lt;ref&gt;However, with a change to the underlying RDF triples, the first encounter also has the possibility to change.&lt;/ref&gt;. 

To create the most precise “approximation”, the equivalent SPARQL 1.1 query must be defined with an embedded SELECT statement. The latter returns the first occurrence of a node (which is accomplished with the MIN(...) binding). Then, we must guarantee that we return a single predecessor (because again, there could be duplicates) by using SAMPLE. The situation is further complicated because we need to return a single predecessor from the minimum depth. This is accomplished by adding a FILTER constraint to force the depth at which the predecessor is selected to be the same as the minimum depth. 

Restating the above, Blazegraph BFS picks one predecessor per visited vertex (the one that first caused the visit). But, the SPARQL SAMPLE predecessor returns one of any of “the predecessors at minimum depth&quot;. This is valid as first-discovery semantics, but may differ from Blazegraph's choice when multiple parents tie at the minimum depth. 

Taking all this into consideration, the final SPARQL query is written as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?depth (SAMPLE(?pred) AS ?predecessor) WHERE {
    { SELECT ?vertex (MIN(?d1) AS ?depth) WHERE {
        {{ BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d1) }
         UNION { wd:Q9682 wdt:P40 ?vertex . BIND(1 AS ?d1) }
         UNION { wd:Q9682 wdt:P40/wdt:P40 ?vertex . BIND(2 AS ?d1) }
         UNION { wd:Q9682 wdt:P40/wdt:P40/wdt:P40 ?vertex . BIND(3 AS ?d1) }}
      } GROUP BY ?vertex          # Get minimum depth per vertex
    }
    {{ BIND(wd:Q9682 AS ?vertex). BIND (0 AS ?d2). }
     UNION { wd:Q9682 wdt:P40 ?vertex . BIND(1 AS ?d2) . BIND(wd:Q9682 AS ?pred) }
     UNION { wd:Q9682 wdt:P40 ?v1 . ?v1 wdt:P40 ?vertex . 
                    BIND(2 AS ?d2) . BIND(?v1 AS ?pred) }
     UNION { wd:Q9682 wdt:P40/wdt:P40 ?v2 . ?v2 wdt:P40 ?vertex .
                    BIND(3 AS ?d2) . BIND(?v2 AS ?pred) }
    }      # Get all paths 3 levels deep
    FILTER(?d2 = ?depth)      # Filter paths that are not the minimum depth
  } GROUP BY ?vertex ?depth ORDER BY ?depth
&lt;/syntaxhighlight&gt;

Which returns the same 27 values in the same order.

Given the complexity of this discussion, it is worthwhile to include rewrite details when the traversal direction is “Reverse”. When the GAS query is reversed, it is especially valuable to include a maxIterations limit (let’s say 3 for ease of rewriting the queries) to bound the results. This makes sense when considering the intent of the query. In the forward direction, the request is for the children of Queen Elizabeth II (a small, bounded group). But, in the reverse direction, it asks about Elizabeth II’s complete ancestry path (not a small group!). 

When executed, there are 15 results up to depth 3.

When rewritten, the query can take one of two forms depending on whether the inverse property path “^” is used, or the order of the subject/object values of the triples are reversed. 

For the first case (using “^”), the query is the same, but wdt:P40 is replaced by ^wdt:P40. Also, the same 15 results are obtained.

In the second case (reversing the subject/object variables), the query becomes:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?depth (SAMPLE(?pred) AS ?predecessor) WHERE {
 { SELECT ?vertex (MIN(?d1) AS ?depth) WHERE {
     {  { BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d1) }
      UNION { ?vertex wdt:P40 wd:Q9682 . BIND(1 AS ?d1) }
      UNION { ?vertex wdt:P40/wdt:P40 wd:Q9682 . BIND(2 AS ?d1) }
      UNION { ?vertex wdt:P40/wdt:P40/wdt:P40 wd:Q9682 . BIND(3 AS ?d1) } }
   } GROUP BY ?vertex
 }
 {  { BIND(wd:Q9682 AS ?vertex) . BIND(0 AS ?d2) }
  UNION { ?vertex wdt:P40 wd:Q9682 .
          BIND(1 AS ?d2) . BIND(wd:Q9682 AS ?pred) }
  UNION { ?vertex wdt:P40 ?v1 . ?v1 wdt:P40 wd:Q9682 .
          BIND(2 AS ?d2) . BIND(?v1 AS ?pred) }
  UNION { ?vertex wdt:P40 ?v1 . ?v1 wdt:P40 ?v2 . ?v2 wdt:P40 wd:Q9682 .
          BIND(3 AS ?d2) . BIND(?v1 AS ?pred) }
 }
 FILTER(?d2 = ?depth)
} GROUP BY ?vertex ?depth ORDER BY ?depth
&lt;/syntaxhighlight&gt;

If this also seems equivalent, look more closely. The ''variable ?v1 is bound as the predecessor at all depths of 2 or more''! This seems to make no sense until you consider the semantics. At depth 3:
* With ^: The predecessor is ?v2, where the chain is Q9682 → v1 → v2 → vertex and BFS depths are 0, 1, 2, 3.
* Without ^: The chain is vertex → v1 → v2 → Q9682 and BFS depths run 3, 2, 1, 0. The predecessor of the ?vertex node is always ?v1 (and this is true for all depths &gt;= 3).

It is valuable to note that (typically) the second syntax (reversing the subject/object) is a slightly more performant query than using the inverse property path (“^”). However, remembering to consider the direction of traversal is a major headache for the query author.  

When performing the rewrite using QLever’s pathSearch, there is a similar need for MIN() and SAMPLE() since a node can have several distinct parents among its paths.

Using pathSearch, the “forward” direction query is structurally similar to the SPARQL 1.1 definition:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
PREFIX pathSearch: &lt;https://qlever.cs.uni-freiburg.de/pathSearch/&gt;
SELECT DISTINCT ?vertex ?depth (SAMPLE(?pred) AS ?predecessor) WHERE {
   { SELECT ?vertex (MIN(?d1) AS ?depth) WHERE {
       { SERVICE pathSearch: {
           _:p pathSearch:algorithm pathSearch:allPaths ;
                 pathSearch:source wd:Q9682 ;
                 pathSearch:maxDepth 3;  
                 pathSearch:start ?pred ; 
	    pathSearch:end ?vertex ;
                 pathSearch:pathColumn ?pc ; 
	    pathSearch:edgeColumn ?edge .
            { SELECT * WHERE { ?pred wdt:P40 ?vertex . } } }
         BIND(?edge + 1 AS ?d1) 
      }
      UNION { BIND(wd:Q9682 AS ?vertex) BIND(0 AS ?d1) }
    } GROUP BY ?vertex }
   { { SERVICE pathSearch: {
         _:p pathSearch:algorithm pathSearch:allPaths ;
               pathSearch:source wd:Q9682 ;
               pathSearch:maxDepth 3; 
               pathSearch:start ?pred ; 
	  pathSearch:end ?vertex ;
               pathSearch:pathColumn ?pc ; 
	  pathSearch:edgeColumn ?edge .
          { SELECT * WHERE { ?pred wdt:P40 ?vertex . } } }
       BIND(?edge + 1 AS ?d2) 
    }
    UNION { BIND(wd:Q9682 AS ?vertex) BIND(0 AS ?d2) } }  
  FILTER(?d2 = ?depth)
} GROUP BY ?vertex ?depth ORDER BY ?depth
&lt;/syntaxhighlight&gt;

There are four items to note: 
* Reporting the predecessor is “free” in pathSearch since it is the pathSearch:start variable
* It is not necessary to explicitly define intermediate nodes (Q9682 → ?v1 → ?v2 → …) because pathSearch carries each edge's source
* The starting/source node has no predecessor in the GAS service results. Therefore,  the depth-0 UNION branch only binds the variables, ?out and ?d2, but leaves the ?pred variable unbound; SAMPLE(?pred) yields null for Q9682
* Although the above is complex, both SERVICE pathSearch definitions are exactly the same, making rewrite straightforward

As an added benefit, there is no need to worry about the “reverse” query since it is constructed by switching the subject and object variables in the nested subquery (?vertex wdt:P40 ?pred).

== Considering GAS Target ==
An additional, possible parameter to the BFS query is “gas:target”. This addition returns only those paths that include the “target” node(s) - leading ''from'' the ‘target’ ''back to'' the “in”/starting node.

When rewriting the query in SPARQL 1.1, it is not possible to define a traversal path from a source to a specific target. This is because it is not possible to stop a query execution at a target. Searching for a target (or multiple targets) has to be performed after the query has completed and returned its results.

Filtering for a specific target relies entirely on the predecessor results. This behavior must be done on the client-side by following these steps:
# Execute the rewritten query specified above.
# Check whether the TARGET node appears in the results.
# If yes, start at the TARGET.
# Repeatedly follow the predecessor nodes until you reach the starting node. 
# Keep only those vertices.

For example, if you have a result that appears as:
::{| class=&quot;wikitable&quot;
|-
! ?vertex !! ?depth !! ?predecessor
|-
| A || 0 ||  
|-
| B || 1 || A
|-
| C || 1 || A
|-
| D || 2 || B
|-
| E || 2 || C
|-
| F || 3 || D
|-
| G || 4 || F
|}

If gas:target is F, the path to the target is:  A -&gt; B -&gt; D -&gt; F, and the reported nodes would be A, B, D, F. 

Working backwards from F in a SPARQL result set, starts at the ?vertex variable = F, ?predecessor = D. Then, ?vertex = D, ?predecessor = B. Then, ?vertex = B, ?predecessor = A (the starting node).

Note that multiple gas:targets can be defined. If this is requested, then the 5 steps above need to be repeated for each target. 

However, performing this on QLever is straightforward. The pathSearch:target parameter is used. It performs the same role as gas:target plus gas:maxIterationsAfterTarget, stopping the search when the target endpoint is reached (if the target is encountered within the specified maxIterations/maxDepth limit).

== Multiple Frontiers ==
GAS’ “frontier” is the set of nodes that participate in the next round of gather/apply/scatter steps. Basically, it is the algorithm’s working set. GAS’ ''initial'' frontier(s) are the node(s) that are defined using the service parameter, gas:in. Note that there may be more than one.  

Examining a sample Blazegraph query using the gas:service with multiple frontiers:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?out ?depth WHERE {
    SERVICE gas:service {
      gas:program gas:gasClass &quot;com.bigdata.rdf.graph.analytics.BFS&quot; ;
                  gas:in wd:Q9682 ;        # Where to start
                  gas:in wd:Q42 ;        
                  gas:linkType wdt:P40 ;   # What property to follow 
                  gas:out ?out ;           # The next vertex (FORWARD traversal)
                  gas:out1 ?depth }
 } ORDER BY ?depth
&lt;/syntaxhighlight&gt;

This query returns the following 29 results:
::[[File:Rewrite for Blazegraph GAS Service - Fig 3.png|Figure 3]]

Reviewing the table, one sees that it is useful for retrieving a list of all descendents, but does not provide any information on the family tree (the starting node) to which the descendent belongs. This is important when there are multiple starting nodes. Also, the query results do not indicate if a descendent occurs in both trees since GAS returns only the first encounter. 

Adding a ?predecessor variable helps a bit, but past the first generation still requires work to assemble the complete tree. 

Rewriting the query as follows provides an equivalent query and also indicates the initial/starting node: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?vertex ?out (MIN(?d) AS ?depth) WHERE {
    VALUES ?vertex { wd:Q9682 wd:Q42 }             # Specifying the gas:in seeds
    {{ VALUES ?vertex { wd:Q9682 wd:Q42 } BIND(?vertex AS ?out) . BIND(0 AS ?d) }
     UNION { ?vertex wdt:P40 ?out . BIND(1 AS ?d) }
     UNION { ?vertex wdt:P40/wdt:P40 ?out . BIND(2 AS ?d) }
     UNION { ?vertex wdt:P40/wdt:P40/wdt:P40 ?out . BIND(3 AS ?d) }}
} GROUP BY ?vertex ?out ORDER BY ?depth
&lt;/syntaxhighlight&gt;

Note that it is still necessary to limit and explicitly define the query depth. 

It is important to note that the VALUES statement is repeated in the query since a variable within a UNION branch cannot see a binding/VALUES for it that is defined outside the UNION. The variable must be brought into the branch's scope.

A portion of the result set (29 members) is shown below: 
::[[File:Rewrite for Blazegraph GAS Service - Fig 4.png|Figure 4]]

Note that the query rewrite above gives additional results since it reports per-seed values. So, a descendant reachable from both of the seeds (Q42 and Q9682) would appear twice, once per seed. (Given that the seeds are Elizabeth II and Douglas Adams, this is very unlikely.) GAS’ results would merge the trees and report each descendant once with one of the predecessors. In this case, the rewrite intentionally produces a more-informative per-seed output. 

Similarly, this query can be written using QLever’s pathSearch. The rewrite is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
PREFIX pathSearch: &lt;https://qlever.cs.uni-freiburg.de/pathSearch/&gt;
SELECT ?vertex ?out (MIN(?d) AS ?depth) WHERE {
  { SERVICE pathSearch: {
       _:p pathSearch:algorithm  pathSearch:allPaths ;
             pathSearch:source wd:Q9682 ;       # Two sources
             pathSearch:source wd:Q42 ; 
             pathSearch:maxDepth 3; 
             pathSearch:start  ?vertex ;     
             pathSearch:end ?out ;
             pathSearch:pathColumn ?pc ; 
             pathSearch:edgeColumn ?edge .
       { SELECT * WHERE { ?vertex wdt:P40 ?out . } } }
    BIND(?edge + 1 AS ?d) }
  # Each source at depth 0
  UNION { VALUES ?vertex { wd:Q9682 wd:Q42 }
                 BIND(?vertex AS ?out) . BIND(0 AS ?d) }
} GROUP BY ?vertex ?out ORDER BY ?depth
&lt;/syntaxhighlight&gt;

It returns the same 29 results.

== Query Rewrite Patterns ==
Across the four GAS algorithms (BFS, SSSP, CC, PR) analyzed in this document, only one (BFS) results in an actual GAS-to-SPARQL rewrite. The other algorithms either reduce to BFS, or have no usable starting query that warrants rewriting. 

At this time, '''only BFS and SSSP queries will be rewritten and tested. Rewrites will utilize QLever’s pathSearch service.'''

The approach to rewriting will be based on the 1:1 correspondence of service parameters that was presented on page 5, along with always capturing the minimum depth as a sub-query, and doing a SAMPLEing of predecessors at the minimum depth in the output variables. 

The only other consideration is whether the gas:in/seed is returned. It is always returned unless the GAS query explicitly removes it by FILTERing for depth &gt; 0. If the latter is true, then the BINDing of the source QID as the pathSearch:start variable, at depth = 0, is unnecessary.

== Footnotes ==</text>
      <sha1>2o5r62uhq6rpp0qi39pdqrji36b0kbr</sha1>
    </revision>
  </page>
  <page>
    <title>Blazegraph Migration: Rewrite of Geospatial Services and Functions</title>
    <ns>0</ns>
    <id>460375</id>
    <revision>
      <id>2433186</id>
      <parentid>2432944</parentid>
      <timestamp>2026-07-05T02:44:48Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>lang=&quot;sparql&quot;</comment>
      <origin>2433186</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="22267" sha1="8hutxcjkdutd4u2o7fwjtn9yro22g64" xml:space="preserve">Geospatial queries are currently suppored in WDQS using Blazegraph-specific SPARQL extensions. There are two SERVICE extensions - wikibase:around (proximity search) and wikibase:box (bounding-box search) - and a small set of helper functions (geof:globe, geof:latitude, geof:longitude). These have no equivalent in SPARQL but can be rewritten using GeoSPARQL.  

Describing each of the concepts in more detail:

* wikibase:box finds entities whose coordinate lies within an axis-aligned rectangle defined by two opposite corners
** Service parameters define either the SouthWest + NorthEast corners of the box (wikibase:cornerSoutWest and wikibase:cornerNorthEast) OR the east/west longitudes (wikibase:cornerWest and wikibase:cornerEast)
*** Determining the box when just longitudes are defined using the following algorithm: 
**** Western edge = longitude of cornerWest 
**** Eastern edge = longitude of cornerEast
**** Southern edge = minimum latitude of cornerWest, cornerEast
**** Northern edge = maximum latitude of cornerWest, cornerEast
*** The southern and northern edges must be calculated in the rewrite, since they are needed to define a box that crosses the ±180° meridian 
** An additional parameter can be used to define the wikibase:globe for non-Earth coordinates
* wikibase:around finds entities whose coordinates lie within a radius of a center point
** The first line of the service request defines a triple, ?place_var predicate_indicating_location ?location_var (the property names are defined by the query author)
*** Note that the predicate can be any that resolves to a globe-coordinate system - for example, P625 is used to indicate a “coordinate location” for the subject entity 
** Service parameters are wikibase:center, wikibase:radius, wikibase:globe and wikibase:distance (where the globe is the Earth by default and the radius is defined in kilometers)
* wikibase:globe reads a point's coordinate system from its WKT literal
* wikibase:latitude and  wikibase:longitude define or read latitude and longitude coordinates for a point

Note that Blazegraph also implements the GeoSPARQL geof:distance function to calculate the distance between two coordinates. It uses a units of kilometers. 
This is translated exactly as-is.

== Query Rewrite Patterns ==

=== POINT Decomposition ===
Blazegraph exposes 3 functions to get geometry details from a point - wikibase:globe, :latitude and :longitude. These can appear ''anywhere'' in a query (SELECT, BIND, FILTER, etc.) or inside wikibase:around/:box blocks. 

This same information can be obtained directly from a GeoSPARQL geo:wktLiteral&lt;ref&gt;A specialized RDF data type used to serialize geometric shapes by binding Well-Known Text (WKT) coordinates to a coordinate system&lt;/ref&gt; for a POINT&lt;ref&gt;Written as “Point” (mixed case) in Wikidata&lt;/ref&gt;. The literal is a string with the format, “[coordinate_reference_system&gt;] POINT(longitude latitude)”. The reference system may be omitted - and, if omitted, defaults to Earth (&lt;http://www.opengis.net/def/crs/OGC/1.3/CRS84&gt;).  

In any case, to extract the coordinate system (wikibase:globe, and latitude/longitude), it is necessary to search within the wktLiteral text:
* Within angle brackets for the coordinate system
* Within parentheses for the longitude/latitude - separated by a space, where the longitude is first

There is a problem, however. When submitting a fully formed wktLiteral (e.g., one that includes the coordinate system), the coordinate specification is ignored by QLever. Worse, including it causes a silent error (zero results are returned). This has been reported as an issue&lt;ref&gt;Issue reported as https://github.com/ad-freiburg/qlever/issues/2982&lt;/ref&gt; on the QLever GitHub site. Support for non-Earth coordinate systems is required and is a work item on QLever’s to-do list.

Note that another option for Earth coordinates&lt;ref&gt;At this time, minX and minY are usable only for the Earth. When non-Earth coordinates are supported, this restriction should be removed.&lt;/ref&gt; is to use the GeoSPARQL geof:minX function to return the longitude of a point, and use the geof:minY function for the latitude. This is the recommended rewrite pattern (versus parsing the wktLiteral).

Also, another option exists to obtain the coordinate system - using GeoSPARQL’s getSRID() function. However, that function is not currently supported by QLever&lt;ref&gt;Issue reported as https://github.com/ad-freiburg/qlever/issues/2984&lt;/ref&gt;. To resolve the specific coordinate system of a point, it is necessary to parse the wktLiteral (as noted above, manually extracting the coordinate system from within the angle brackets of the wktLiteral).

=== wikibase:around ===
A rewrite of wikibase:around replaces the service request with the search details (center, radius and distance) as follows:

* As noted in the Overview, the first line of the :around service is `?place_var ⟨location_predicate⟩ ?location_var`, where the location_predicate is a property IRI which resolves to an object with a geo:wktLiteral data type (e.g., P625)
** This line is extracted and duplicated in the rewrite “as-is”
* bd:serviceParam’s wikibase:center, :radius and :distance names/values are mapped as follows:
** wikibase:center is either a specific POINT geometry or variable having a wktLiteral data type
*** It is mapped “as defined” into a BIND statement using the geof:distance function (it is geof:distance’s first parameter)
*** The second parameter of geof:distance is the ?location_var from the first line
** wikibase:radius is either a constant or a variable identifying a numeric value
*** Its units are kilometers and is rewritten exactly as defined
** wikibase:distance is always a variable name that is bound when the service request is evaluated
** The resulting SPARQL 1.1 compliant form is: `BIND(geof:distance(?center_var_name_or_POINT_value, ?location_var, unit:KiloM) AS ?distance_var_name)`
*** Since the default units for geof:distance are kms, the geof:distance’s third argument can be removed 
** Plus `FILTER(?distance_var_name &lt;= ?radius_var_name_or_constant)`
* bd:serviceParam’s wikibase:globe cannot currently be supported by QLever as discussed above
** At present, only an Earth coordinate system is valid
** The rewrite code will correctly process non-Earth coordinate systems, although those queries will fail on QLever until an update is available

==== Example ====
The following query returns all airports in a 100km radius around Berlin.
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?place ?location ?dist 
WHERE {
   wd:Q64 wdt:P625 ?berlinLoc .     # Berlin coordinates
   SERVICE wikibase:around { 
       ?place wdt:P625 ?location . 
       bd:serviceParam wikibase:center ?berlinLoc . 
       bd:serviceParam wikibase:radius &quot;100&quot; . 
       bd:serviceParam wikibase:distance ?dist.
   } 
   FILTER EXISTS {
       # Is an airport
       ?place wdt:P31/wdt:P279* wd:Q1248784 .
   } 
} ORDER BY ASC(?dist)
&lt;/syntaxhighlight&gt;

Which returns the following results:
[[File:Geospatial Rewrite-Fig 1.png|Figure 1]]

The rewritten query is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?place ?location ?dist 
WHERE {
   wd:Q64 wdt:P625 ?berlinLoc . 
   ?place wdt:P625 ?location .          # Extracted first line
   BIND(geof:distance(?berlinLoc, ?location) AS ?dist)
   FILTER(?dist &lt;= 100)    
   FILTER EXISTS {
      # Is an airport
      ?place wdt:P31/wdt:P279* wd:Q1248784 .
   } 
} ORDER BY ASC(?dist)
&lt;/syntaxhighlight&gt;

Which returns the results:
[[File:Geospatial Rewrite-Fig 2.png|Figure 2]]

Note that although both queries return 7 results, there is a difference in precision and distance values.

When dealing with geospatial calculations, differences in values may result. For example, if requesting the distance from Berlin to Paris, the following is reported:
* Blazegraph - 875.9227km
* QLever - 878.3542km

This is a 2.43 km difference (~0.28%). An issue has been raised on the QLever GitHub site related to this&lt;ref&gt;Issue reported as https://github.com/ad-freiburg/qlever/issues/2989&lt;/ref&gt;.

=== wikibase:box ===
wikibase:box specifies a rectangular region using latitude and longitude and ascertains whether a location is within that region. Using SPARQL 1.1 and GeoSPARQL, there are two ways to rewrite the query. Choosing one over the other depends on whether the region includes the meridian (180°0'E or 180°0'W) and whether the user wants to include locations that fall on the meridian.   

It is interesting to note that including locations on the meridian results in different answers for QLever and Blazegraph. See the second example below for details. 

Rewriting the query is defined by the lat/long details supplied in the service request, with the addition of the coordinate system. Similar to wikibase:around, a non-Earth coordinate system is not supported. But, as this work item is addressed by the QLever team, updates will be made to this document and the rewrite code. 

Converting the query is handled similarly to wikibase:around, replacing the service request by:
* Extracting and copying the first line
* Deriving the “bounds” (east/west longitudes and north/south latitudes) from the service details
** For the cornerSouthWest and cornerNorthEast parameters, southern/northern-most and western/eastern-most coordinates are obvious
*** The geof:minX/minY functions are used to extract the north-south-east-west coordinates
*** A POLYGON shape is constructed from these coordinates
**** It must define a closed loop (SW → SE → NE → NW → SW)
** For the cornerWest and cornerEast parameters (the region crosses the meridian), only the east/west longitudes are defined and the north/south latitudes must be determined from the specific POINT data using min/max comparison
*** The geof:minX/minY functions are used to extract the north-south-east-west coordinates and then a SPARQL IF statement selects the min/max latitudes
* Checking for locations within the region using a FILTER to guarantee presence within the latitude/longitude boundaries
** The geof:sfContains() function indicates whether the location (the variable identified as the second parameter)  is within the bounding region
*** sfContains is used if the region does not cross the meridian or if the locations on the meridian can be excluded
** If points on the meridian are to be returned in the query results (which is reasonable when they are within the bounding region), then an explicit coordinate-range FILTER is defined
*** The query rewrite uses this method for rewrites since it is the most complete
* As above, bd:serviceParam’s wikibase:globe cannot currently be supported by QLever

==== Example 1 ====
The following query checks for schools with the region defined in the service request. It is ordered by the school name for ease of comparison.
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?place ?placeLabel ?location WHERE {
   wd:Q16553 wdt:P625 ?SJloc.
   wd:Q18013 wdt:P625 ?SCloc.
   SERVICE wikibase:box {
       ?place wdt:P625 ?location.
       bd:serviceParam wikibase:cornerSouthWest ?SJloc.
       bd:serviceParam wikibase:cornerNorthEast ?SCloc.
   }
   FILTER EXISTS { ?place wdt:P31/wdt:P279* wd:Q3914. }    # A school
   SERVICE wikibase:label { bd:serviceParam wikibase:language &quot;en&quot;. } 
} ORDER BY ?placeLabel
&lt;/syntaxhighlight&gt;

Here are some of the results:
[[File:Geospatial Rewrite-Fig 3.png|Figure 3]]

The rewritten query is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?place ?placeLabel ?location WHERE {
    wd:Q16553 wdt:P625 ?SJloc .          # San Jose   -&gt; south-west corner
    wd:Q18013 wdt:P625 ?SCloc .          # Sacramento -&gt; north-east corner

    # Build the box as a single POLYGON from the SW/NE corners 
    # minX is longitude, minY is latitude
    BIND(geof:minX(?SJloc) AS ?w)  BIND(geof:minY(?SJloc) AS ?s)  
    BIND(geof:minX(?SCloc) AS ?e)  BIND(geof:minY(?SCloc) AS ?n)
    BIND(STRDT(CONCAT(&quot;POLYGON((&quot;, 
         STR(?w),&quot; &quot;,STR(?s), &quot;,&quot;, STR(?e),&quot; &quot;,STR(?s), &quot;,&quot;,
         STR(?e),&quot; &quot;,STR(?n), &quot;,&quot;, STR(?w),&quot; &quot;,STR(?n), &quot;,&quot;,
         STR(?w),&quot; &quot;,STR(?s), &quot;))&quot;), geo:wktLiteral) AS ?box)
    ?place wdt:P625 ?location .                       # Extracted first line
    FILTER(geof:sfContains(?box, ?location))          # sfContains test

    # EXISTS in the original query and maintained as a separate clause due to QLever bug
    FILTER EXISTS { ?place wdt:P31/wdt:P279* wd:Q3914 . }  
 
    # wikibase:label -&gt; explicit label with Q-ID fallback
    OPTIONAL { ?place rdfs:label ?plLabel . FILTER(LANG(?plLabel) = &quot;en&quot;) }
    BIND(COALESCE(STR(?plLabel), STRAFTER(STR(?place), &quot;entity/&quot;)) AS ?placeLabel)
} ORDER BY ?placeLabel
&lt;/syntaxhighlight&gt;

Which returns the same results.

==== Example 2 ====
This next example is a query where only the longitudes are defined because the desired “box” crosses the meridian. Because longitudes wrap at the meridian, any two longitudes can define two paths - the long way (around the globe) and the short way (crossing the meridian).

Using the cornerSouthWest and cornerNorthEast parameters, the result is the “long way”. For example, consider cornerSouthWest=(170,−25) and cornerNorthEast=(−170,−10). The Blazegraph wikibase:box service computes the longitude interval as [min(170,−170), max(170,−170)] = [−170, 170]. This is a 340°-wide band running through longitude 0°. But, when defining cornerWest and cornerEast, the service instead always moves west to east. Since 170 &gt; −170, the service knows to cross ±180° (the correct 20° band over the date line). 

The following query searches for all islands in the South Pacific bounded by Fiji, Tonga and Samoa.

&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?place ?loc WHERE {
   SERVICE wikibase:box {
     ?place wdt:P625 ?loc .
     bd:serviceParam wikibase:cornerWest &quot;Point(170 -25)&quot;^^geo:wktLiteral .
     bd:serviceParam wikibase:cornerEast &quot;Point(-170 -10)&quot;^^geo:wktLiteral .
   }
   ?place wdt:P31 wd:Q23442 .          # islands
}
&lt;/syntaxhighlight&gt;

Shown below is a portion of the 891 results.
::[[File:Geospatial Rewrite-Fig 4.png|Figure 4]]

It is important to note that Blazegraph returns 891 results despite the fact that 3 islands lie on the meridian. Two of the 3 islands have  a 180°0'E longitude and are included. The one island with a 180°0'W longitude (Rabi Island, Q762070) is excluded.

The closest approximation to Blazegraph’s behavior is to manually create the bounding polygon in the FILTER. This can be accomplished as follows:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?place ?loc WHERE {
   BIND(&quot;Point(170 -25)&quot;^^geo:wktLiteral  AS ?cornerW)   # from cornerWest
   BIND(&quot;Point(-170 -10)&quot;^^geo:wktLiteral AS ?cornerE)   # from cornerEast
  
   BIND(geof:minX(?cornerW) AS ?west)        # west longitude
   BIND(geof:minX(?cornerE) AS ?east)        # east longitude
   BIND(geof:minY(?cornerW) AS ?latW)       
   BIND(geof:minY(?cornerE) AS ?latE)
   BIND(IF(?latW &lt; ?latE, ?latW, ?latE) AS ?south)     
   BIND(IF(?latW &lt; ?latE, ?latE, ?latW) AS ?north)

   ?place wdt:P625 ?loc .           # Extracted first line
   BIND(geof:minX(?loc) AS ?locLong)
   BIND(geof:minY(?loc) AS ?locLat)
  
   # place’s location’s latitude &gt;=?south, and &lt;= ?north
   FILTER(?locLat &gt;= ?south &amp;&amp; ?locLat &lt;= ?north)
   # place’s location’s longitude &gt;= ?west &amp; &lt;= ?east; Account for wrapping the meridian
  FILTER( ( ?west &lt;= ?east &amp;&amp; ?locLong &gt;= ?west &amp;&amp; ?loc &lt;= ?east ) ||
            ( ?west &gt;   ?east &amp;&amp; (?locLong) &gt;= ?west || ?locLong &lt;= ?east ) )	
	  
  FILTER EXISTS { ?place wdt:P31 wd:Q23442 }       # islands 
}
&lt;/syntaxhighlight&gt;

Note, however, that 892 results are returned. Executing this query on QLever correctly returns all 3 points on the meridian. 

It is valuable to note that the query above can be further simplified to move the triple pattern within FILTER EXISTS to the outer graph.  Since the pattern does not involve an arbitrary-length property path, but is a direct triple, the bug mentioned in Footnote 7 is not encountered. However, the rewrite code does not make this change since a one-to-one mapping of the query (replacing the Blazegraph-specific functions) is the goal. Further tuning of the query is left to the user.

Another rewrite possibility exists for the query - where all points on the meridian are excluded from the results. (This is mentioned since it is the algorithm that Blazegraph uses.) The query is rewritten to define the region of interest as MULTIPOLYGON. The latter is the union of two (or more) polygons. For this case, it is the union of a polygon defined on each side of the meridian. The query appears as follows:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?place ?loc WHERE {
    BIND(&quot;Point(170 -25)&quot;^^geo:wktLiteral  AS ?cornerW)   # cornerWest
    BIND(&quot;Point(-170 -10)&quot;^^geo:wktLiteral AS ?cornerE)   # cornerEast

    BIND(geof:minX(?cornerW) AS ?west)  BIND(geof:minX(?cornerE) AS ?east)   
    BIND(geof:minY(?cornerW) AS ?latW)  BIND(geof:minY(?cornerE) AS ?latE)
    BIND(IF(?latW &lt; ?latE, ?latW, ?latE) AS ?south)       
    BIND(IF(?latW &lt; ?latE, ?latE, ?latW) AS ?north)

    # Wrapping box -&gt; a MULTIPOLYGON split at ±180
    #  eastern ring:  west south , 180  south , 180  north , west north , west south
    #  western ring:  -180 south , east south , east north , -180 north , -180 south
    BIND(STRDT(CONCAT(
         &quot;MULTIPOLYGON(((&quot;, 
             STR(?west),&quot; &quot;,STR(?south),&quot;,&quot;, &quot;180 &quot;,STR(?south),&quot;,&quot;,
             &quot;180 &quot;,STR(?north),&quot;,&quot;, STR(?west),&quot; &quot;,STR(?north),&quot;,&quot;, 
             STR(?west),&quot; &quot;,STR(?south),
             &quot;)),((&quot;, 
              &quot;-180 &quot;,STR(?south),&quot;,&quot;, STR(?east),&quot; &quot;,STR(?south),&quot;,&quot;, 
              STR(?east),&quot; &quot;,STR(?north),&quot;,&quot;, &quot;-180 &quot;,STR(?north),&quot;,&quot;, 
              &quot;-180 &quot;,STR(?south), &quot;)))&quot;),
         geo:wktLiteral) AS ?box)

    ?place wdt:P625 ?loc .
    FILTER(geof:sfContains(?box, ?loc))
   
    FILTER EXISTS { ?place wdt:P31 wd:Q23442 }       # islands 
}
&lt;/syntaxhighlight&gt;
             
This query returns 889 results since the 3 islands defined on the meridian (with a longitude of 180°0'E or 180°0'W) are excluded. And, as noted above, the triple within the FILTER EXISTS clause could be moved to the outer graph.

== QLever GeoSPARQL Compliance ==
QLever is partially compliant with the two GeoSPARQL conformance classes relevant to Wikidata - Geometry and Geometry Topology. 

The bullets below describe all 7 of the GeoSPARQL conformance classes, their relevance to Wikidata and corresponding QLever support. 
* Core Vocabulary (N/A) - The vocabulary defines the classes, geo:SpatialObject and geo:Feature, and their subclass relationships
** These distinctions are not relevant in the WDQS environment since location is specified by predicates such as wdt:P625 (coordinate location)
* Topology Vocabulary (N/A) - The vocabulary defines the geo:sf* predicates which can be used in triple data 
** These predicates are not present in Wikidata, but may be returned by federated services such as OpenStreetMap data
** The predicates are handled by QLever if present in the data
* Geometry (Partial compliance) - Establishes WKT (Well-Known Text serialization), defines the concepts and properties needed to represent spatial shapes like points, lines, and polygons, and provides GeoSPARQL functions to query geometric attributes such as area, length, distance, and a coordinate reference system
** QLever supports the following functions: geo:asWKT, geof:distance, :metricDistance, :length, :geometryType, :min/maxX/Y, :envelope, :centroid, :area and :metricArea 
** There is no support for the functions, geof:asGML/asGeoJSON/asKML, :buffer, :metricBuffer, :convexHull, :concaveHull, :boundingCircle, boundary, :union, :intersection, :difference, :symDifference, :perimeter, :metricPerimeter, :getSRID, :relate, :dimension, :coordinateDimension, :spatialDimension, :isEmpty, :isSimple, :is3D, :min/max/Z/M, :isMeasured or :transform
** At this time, there is no QLever support for non-Earth coordinate systems or for adding a coordinate system to a WKT
* Geometry DGGS, Discrete Global Grid System (N/A) - An alternate mechanism for representing location (via a grid)
** Not relevant for WDQS
* Geometry Topology (Partial compliance) - Provides specific vocabulary and functions to evaluate topological relationships between spatial objects, analyzing how geometries intersect, overlap, touch, or contain one another 
** QLever provides the Simple-Features geof:sf* functions except for geof:sfDisjoint and the functions of the Egenhofer/RCC8 families
** QLever also provides a geof:sfCovers function, which is not in the GeoSPARQL standard&lt;ref&gt;&quot;Covers&quot; exists only in the Egenhofer family, as geof:ehCovers&lt;/ref&gt; 
** In QLever, sf* functions evaluate as spatial joins where at least one argument must be a multi-valued, spatially-indexed geometry variable (e.g., ?g in ?x wdt:P625 ?g, ranging over the whole P625 column); the other argument may be a constant or query-constructed geometry (e.g. a box polygon)
** Queries fail when the geometry variable is pinned to one entity (e.g., wd:Q64 wdt:P625 ?g where ?g is a singleton), or when both arguments are constants&lt;ref&gt;This is a small non-compliance issue since geof:sf functions should operate over any two geometry literals, but a geof:distance call can be used instead.&lt;/ref&gt;
* RDFS Entailment (N/A) - Reasoning and inference
** N/A for the WDQS environment
* Query Rewrite (No support) - Automatic translation of queries about geospatial “features” (such as a building or park) into their lower-level geometric calculations 
** Where the Topology Vocabulary defines the relation properties - geo:sfContains, geo:sfWithin, etc. - as RDF predicates, Query Rewrite allows computation of those relations from the default geometries of the “features”
** Using QLever, testing if two features overlap requires connecting them to their geometries and then testing the geometries with the Geometry Topology functions (such as geof:sfContains)

Also, it is important to remember that only the wktLiteral serialization is supported. GML (Geography Markup Language), GeoJSON, KML (Keyhole Markup Language)  and DGGS are not supported. And, at present, globe-prefixed WKT literals silently fail. 

== Footnotes ==</text>
      <sha1>8hutxcjkdutd4u2o7fwjtn9yro22g64</sha1>
    </revision>
  </page>
  <page>
    <title>Blazegraph Migration: Rewrite of Label and Utility Services and Functions</title>
    <ns>0</ns>
    <id>460376</id>
    <revision>
      <id>2433187</id>
      <parentid>2431225</parentid>
      <timestamp>2026-07-05T02:47:50Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>lang=&quot;sparql&quot;;  lang=&quot;text&quot;;</comment>
      <origin>2433187</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="32631" sha1="9w0q75leehs373tax0zekr8bjjxqjjv" xml:space="preserve">This Wikitech page catalogs the rewrites needed to translate several utility query services provided by Blazegraph into portable, standards-conformant SPARQL 1.1. To port these queries, each Blazegraph-specific construct must be substituted with an equivalent SPARQL 1.1 pattern, restructured into a portable form, or (where no equivalent exists) removed with an explanatory note. 

The rewrites described in this document cover six such constructs: 
* bd:sample and bd:slice result-limiting predicates
* Named subqueries 
* Inline query hints 
* The wikibase:decodeURI function 
* The wikibase:label service  

For each construct, the sections that follow explain its Blazegraph-specific semantics and specify the rewrite.

== bd:sample ==
bd:sample is a Blazegraph service that selects a bounded subset of solutions from a graph pattern &lt;i&gt;without&lt;/i&gt; fully materializing and sorting the entire set of results. Its purpose is to keep otherwise-expensive queries under the WDQS timeout by working on a sample instead of the full data. 

For example, the following query returns 2000 “locations” (&lt;i&gt;wdt:P276 ?value&lt;/i&gt;) and orders them by the number of distinct ?items that reference them. (Note however, that this is the ordering within the sample set of results and not across Wikidata.)

&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?value (COUNT(DISTINCT ?item) as ?count) WHERE {
 SERVICE bd:sample {
   ?item wdt:P276 ?value .
   bd:serviceParam bd:sample.limit 2000 . }
} GROUP BY ?value
  ORDER BY DESC(COUNT(DISTINCT ?item))
&lt;/syntaxhighlight&gt;

The query pattern defined within the service binds the relevant variables (in this case, ?item and ?value). These can then be used in the surrounding SPARQL.

The possible service parameters are:
{| class=&quot;wikitable&quot;
|-
! Parameter !! Usage !! Default
|-
| bd:sample.limit || Maximum number of result sets to return || 100
|-
| bd:sample.sampleType || “RANDOM”, “EVEN” (evenly spaced sample), “DENSE” (first N) || RANDOM
|-
| bd:sample.seed || RANDOM's seed || 0 
|}

Rewriting the service depends on the sampleType parameter, but none of the rewrites preserve the performance characteristics of bd:sample. For all rewrites, the engine must evaluate the inner pattern first and run to completion. 

The closest SPARQL 1.1 query to that shown above is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?value (COUNT(DISTINCT ?item) AS ?count) WHERE {
  { # Translate the query pattern within the service to a SELECT query
    SELECT ?item ?value WHERE {    
      ?item wdt:P276 ?value .
    } ORDER BY RAND()      # Default is RANDOM, so apply that ordering
      LIMIT 2000           # LIMIT 2000
  }
} GROUP BY ?value
  ORDER BY DESC(COUNT(DISTINCT ?item))
&lt;/syntaxhighlight&gt;

The above query does complete on QLever, but a query that returns a larger result set (for example, asking for a subset of humans, Q5) or is more complex could exceed the 60 second timeout. In all cases, applying a LIMIT to the inner query is needed to force a cut-off. That is also why bd:sample has a limit parameter.

It is valuable to note that the SPARQL 1.1 query could be changed to remove the ORDER BY RAND() statement and just use LIMIT. That would definitely improve the performance, and would mimic choosing the “DENSE” sampleType of bd:sample. 

“EVEN” is the last sampleType value that could be used. To replicate the behavior requires positional/windowing logic that standard SPARQL 1.1 cannot express. However, the result can be approximated with the RANDOM rewrite above (a uniform random sample is usually an acceptable stand-in for an evenly-spaced one).

There is another service parameter that we did not discuss - seed. This would be used for reproducibility. Here is a Blazegraph query using that parameter:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  SERVICE bd:sample {
    ?item wdt:P31 wd:Q5 .                         # ?item is an instance of human
    bd:serviceParam bd:sample.limit 1000 .
    bd:serviceParam bd:sample.sampleType &quot;RANDOM&quot; .
    bd:serviceParam bd:sample.seed 42 .           # fixed seed → same 1000 every run
  }
}
&lt;/syntaxhighlight&gt;

The rewrite is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  { SELECT ?item WHERE {
      ?item wdt:P31 wd:Q5 .
    } ORDER BY MD5(CONCAT(&quot;42&quot;, STR(?item)))   # &quot;42&quot; plays the role of bd:sample.seed
      LIMIT 1000
               }
}
&lt;/syntaxhighlight&gt;

Note that the above query can successfully execute on a QLever server in under 60 seconds. So, simply having a large intermediate result set does not necessarily equate to query timeout.

=== Query Rewrite Pattern for bd:sample ===
These steps summarize the rewrite patterns for bd:sample:
* Lift the query &lt;pattern&gt; out of SERVICE bd:sample {…}  and rewrite it as an inner SELECT … WHERE { &lt;pattern&gt; }, projecting the variables present in the outer query 
* Append LIMIT N (the bd:sample.limit or use the default, 100, if it was omitted)
* Add an ORDER BY if the sampleType is “RANDOM” or “EVEN” 
** RAND() if a seed is not specified
** MD5(CONCAT(seed, STR(?key))) otherwise
* Leave the surrounding query (GROUP BY, COUNT, etc.) unchanged

== bd:slice ==
bd:slice is a Blazegraph service that provides efficient, stable pagination over a single triple pattern. It walks through a very large result set in deterministic chunks without re-sorting on each request. Its purpose is to keep otherwise-expensive queries under the WDQS timeout by working on a subset instead of the full data. 

The most “mechanically” equivalent SPARQL 1.1 pattern is to perform the query with an ORDER BY, and OFFSET / LIMIT. However, this becomes increasingly non-performant as the OFFSET size grows - since OFFSET skips rows in the result set by first generating and then discarding them. 

Another way to rewrite the query using SPARQL 1.1 is to create an “index” and use that to segment the results. Taking this approach each page carries a similar performance cost. 

For example, the following Blazegraph query is used to return result sets of 10000 humans. Subsequent calls increase the offset by 10000 and add 10000 to the limit. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
 SERVICE bd:slice {
   ?item wdt:P31 wd:Q5 .                    # single triple pattern (required)
   bd:serviceParam bd:slice.offset 0 .
   bd:serviceParam bd:slice.limit 10000 .
 }
}
&lt;/syntaxhighlight&gt;

To rewrite this query using SPARQL 1.1 and a query-relevant index requires some manual construction … The largest value of the value from a run is inserted into a FILTER on the index.

For the query above, the first iteration is written as: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .             
} ORDER BY (str(?item))
  LIMIT 10000  
&lt;/syntaxhighlight&gt;

The second iteration becomes: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .
  FILTER(STR(?item) &gt; &quot;http://www.wikidata.org/entity/Q100310806&quot;)
} ORDER BY STR(?item)
  LIMIT 10000 
&lt;/syntaxhighlight&gt;

Where the entity, Q100310806, is the “last” ?item output by the previous iteration.

It is important to note that because of the indexes and performance characteristics of QLever, it is also possible to execute the query unconstrained  by order or limits. 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item WHERE {
  ?item wdt:P31 wd:Q5 .
}  
&lt;/syntaxhighlight&gt;

A total of 13M+ results are output, and indicates that bd:slice was a performance optimization required by Blazegraph as opposed to addressing a query-related need.

As for bd:sample, there is another service parameter for bd:slice that should be discussed - range. In range mode, bd:slice doesn't return rows of results, but returns a single value - a count of the solutions that is computed from index boundaries. A variable is named and bd:slice.range estimates a count. Note that word, estimates. This is problematic since it is usually an overestimate and done as another performance optimization.

Consider these two Blazegraph queries:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?range WHERE {
  SERVICE bd:slice {
    ?item wdt:P569 ?dob .                       # single triple pattern
    bd:serviceParam bd:slice.range ?range .     # ?range ← fast index count of matches
  }
}
&lt;/syntaxhighlight&gt;

And:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?range WHERE {
  SERVICE bd:slice {
    ?item wdt:P569 ?dob .
    FILTER(?dob &gt;= &quot;1900-01-01&quot;^^xsd:dateTime &amp;&amp; ?dob &lt; &quot;2000-01-01&quot;^^xsd:dateTime)
    bd:serviceParam bd:slice.range ?range .
  }
}
&lt;/syntaxhighlight&gt;

Both of these queries return a ?range of approximately 8074600 (varying +/- 3).

However, a corresponding query using SPARQL 1.1, gives exact results.
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT (COUNT(*) AS ?range) WHERE {
  ?item wdt:P569 ?dob .
  FILTER(?dob &gt;= &quot;1900-01-01T00:00:00&quot;^^xsd:dateTime &amp;&amp; 
               ?dob &lt; &quot;2000-01-01T00:00:00&quot;^^xsd:dateTime)
}
&lt;/syntaxhighlight&gt;

Which returns a count of 5457644. With the FILTER removed, the count is 8074572 (pretty close to the overall Blazegraph estimate!).

=== Query Rewrite Pattern for bd:slice ===
The following list defines the rewrite patterns for bd:slice with limit and offset, or with range:
* For limit/offset:
** Evaluate first if limit/offset are actually needed. For many queries, QLever’s indexes and performance are sufficient to remove the need for this optimization.
** If the query does timeout, then pagination is needed. The simplest portable form is to use ORDER BY &lt;key&gt; + OFFSET + LIMIT, bumping OFFSET with each query request. But, note that OFFSET cost grows with depth. 
*** For deep paging, it is recommended to use keyset pagination. Pick a deterministic sort key (e.g. STR(?subject)), ORDER BY it, issue an initial LIMIT query, then for each subsequent page add &lt;i&gt;FILTER(STR(?subject) &gt; “last_value_from_previous_query”)&lt;/i&gt; with the same ORDER BY and LIMIT. 
*** Keyset replaces the growing OFFSET with an index seek
*** Remember - do not use both OFFSET and the cursor FILTER together
** This is accomplished as a manual rewrite based on the bd:slice query pattern with the addition of a cursor/index (such as STR(?subject_var)) and ORDER BY statement using this index. A manual evaluation of the results is required - to determine the largest index value returned for the query. After the first query, each one has to include a FILTER() statement based on the largest index value returned, along with a LIMIT and OFFSET.
* For range:
** Use the query pattern as-is and &lt;i&gt;SELECT (COUNT(*) AS ?range)&lt;/i&gt;

== Named Subqueries ==
Although SPARQL supports nested subqueries, it does not support assigning them a name or specifically “including” them at specific (sometimes multiple) points in a scoping query. Blazegraph allows this using the keywords, INCLUDE, AS and WITH. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer
WITH {
  SELECT ?person WHERE {
    ?person wdt:P31 wd:Q5 ; wdt:P106 wd:Q901 .   # humans who are scientists
  }
} AS %scientists
WHERE {
  INCLUDE %scientists .
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

The query first executes the &lt;i&gt;WITH { … } AS %name&lt;/i&gt; subquery, and does this only once. Its results are materialized into a named temporary set, and every &lt;i&gt;INCLUDE %name&lt;/i&gt; joins against that same cached set. It's effectively a SQL Common Table Expression (CTE) using WITH … AS. 

Named subqueries serve three purposes: 
* Clarity - cleanly defining the query
* Reuse - allowing an expensive subquery to be executed once and used in several places 
* Join-order control - forcing the subquery to run first, rather than letting the query optimizer to insert it

Unfortunately, SPARQL 1.1 has no equivalent to named subqueries. The portable equivalent is an inline sub-SELECT that duplicates the code of the WITH clause and replaces each &lt;i&gt;INCLUDE %name&lt;/i&gt; with the body of the sub-SELECT. For the example above:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer WHERE {
  { SELECT ?person WHERE {
      ?person wdt:P31 wd:Q5 ; wdt:P106 wd:Q901 .
  } }
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

The most important question is whether the subqueries are simply copied wherever they are referenced. And, for ordinary, deterministic queries, the answer is yes. The sub-SELECT body can be pasted at each INCLUDE site&lt;ref&gt;This is actually a worst case rewrite since the pattern can be simplified to only paste within each unique query scope. This is discussed in more detail in the Query Rewrite sub-section.&lt;/ref&gt;.

Again, unfortunately, &quot;simply copied&quot; hides three problems:  
* Complexity, when a sub-query is copied multiple times and the query text “explodes” 
* Performance, since the compute-once guarantee is lost
** Blazegraph evaluates the block once regardless of the number of references to it
** Instead, if it is inlined at N sites, then the query is evaluated N times
** For a “cheap” subquery, this is mainly irrelevant
** For an “expensive” subquery referenced several times, a large performance cost could be incurred
* Correctness, when the sub-query is non-deterministic or mints blank nodes AND occurs multiple times
** Non-deterministic queries include functions such as RAND() and UUID()/STRUUID()
*** When Blazegraph executes the sub-query once, every INCLUDE sees the same bound variables and values
*** Instead, if inlined multiple times, each copy re-evaluates those functions and returns different values
** The sub-query may return variables as blank nodes
*** As above, when Blazegraph executes the sub-query once, every INCLUDE sees the same bound variables and bnode values
*** Instead, if inlined multiple times, each copy returns different bnode identities
** Other examples of non-deterministic results are subqueries that use:
*** A LIMIT but no ORDER BY, making the result selection defined at execution time
*** GROUP_CONCAT, since the SPARQL aggregate functions operate over unordered sets
*** SAMPLE()
*** A federated service call to an external endpoint
**** If there are N separate calls to the federated service, results could vary between the calls due to changes in the data, a LIMIT on the federated result, or other non-deterministic behavior at the endpoint
*** Blazegraph-specific sampling utilities such as bd:slice and bd:sample

If non-deterministic or blank node results are included in the solution set, &lt;i&gt;AND the named subquery is used more than once&lt;/i&gt;, it cannot just be copied but must first be executed/materialized. Then, its results can be manually added back into the query as a VALUES block.

For example, rewriting the scientist query above using the VALUES approach (even though this is deterministic), results in the following: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?person ?employer WHERE {
  VALUES ?person { wd:Q4118502 wd:Q100156581 … }   # Results of running %scientists once
  ?person wdt:P108 ?employer .
}
&lt;/syntaxhighlight&gt;

There are a few obvious problems with the VALUES approach. First, it is a two-step manual or programmatic process. Second, it is cumbersome when the result set is large (thousands of results).

=== Query Rewrite Pattern for Named Subqueries ===
For rewriting a named subquery, its contents (within the WITH { … } AS clause) are replaced by an inlined &lt;i&gt;{ SELECT variables WHERE { … } }&lt;/i&gt; clause. If there is only a single reference to the sub-query, the result of inlining is equivalent to the original query. 

However, questions arise when there are multiple references to the sub-query and these occur in multiple scopes&lt;ref&gt;Examples of different scopes in a SPARQL query are having nested sub-queries or separate UNION branches.&lt;/ref&gt;. One solution to this is to rewrite the sub-query in each scope. Further complications then arise only when the results of the sub-query are non-deterministic and this is visible in the externalized / observable result variables. 

There is complexity in determining these two characteristics (non-deterministic and projected). To determine them, the query’s algebra tree&lt;ref&gt;Obtained using Jena ARQ or rdflib libraries&lt;/ref&gt; must be analyzed. This can be used to first define whether multiple scopes are involved. If the enclosing graph pattern for each reference to a named subquery is the same - then there is only one scope (and no rewrite problems). However, if two INCLUDEs have different enclosing patterns, then their potential results need to be examined. 

A simple way to determine if the results are non-deterministic due to generated blank nodes is to actually run the sub-query and see if any results are reported with type = “bnode”&lt;ref&gt;It is important to distinguish Wikidata skolemized blank nodes, and query engine-minted nodes. Wikidata’s blank nodes will be returned in a query binding as type=&quot;uri&quot; with a specific, consistent IRI value.&lt;/ref&gt;. If so and the subquery is referenced in multiple scopes, then that query is not eligible for rewriting.

Other simple checks for non-deterministic results is whether the subquery includes bd:slice or bd:sample (which have approximate rewrites), or a LIMIT without an ORDER BY. 

Lastly, if the subquery uses SAMPLE(), RAND(), UUID(), STRUUID() or GROUP_CONCAT(), and if that use occurs in multiple scopes, and if the results of the function calls are observable in the results, then the query is &lt;b&gt;ineligible&lt;/b&gt; for rewrite. Being “observable” occurs if the functions appear in the SELECT clause, affect which result sets are returned (used in FILTER/HAVING or ORDER BY under a LIMIT/OFFSET), or feed a projected BIND. 

Examples of these are:
* A SELECT variable (for example, SELECT ?k (SAMPLE(?v) AS ?s))
* A FILTER/HAVING clause (e.g., HAVING(SAMPLE(?v) &gt; 5))
* An ORDER BY (SAMPLE(?v)) ... LIMIT n clause (since the results which “clear” the LIMIT are dependent on the SAMPLE selection) 

=== What about QLever’s Materialized Views? ===
QLever has defined a feature (still in beta at the time of writing) called [https://docs.qlever.dev/materialized-views/ materialized views] which seems similar to Blazegraph’s named subqueries. Both provide the capability to create a named, compute-once result set that is referenced by name. However, there are several differences and explicit issues. The most important of these are: 
* SPARQL 1.1 compliance:
** Neither Blazegraph named subqueries nor QLever materialized views are SPARQL 1.1 compliant
** Also, the creation of a materialized view requires server-level access (not allowed on WDQS)
* Definition location: 
** A named subquery (WITH … AS %x) is defined inside the query itself - it is self-contained and recomputed every run
** A materialized view is created out of band and stored on disk (dependent on server-side state) 
* Lifetime and freshness:
** Since named subqueries are recomputed on every query execution, they are never stale
** Materialized views are persistent and must be recreated if they go stale 

== Query Hints ==
QLever does not expose a Blazegraph-style query-hint system and this is not a part of the SPARQL 1.1 standard. So, the rewrite pattern for Blazegraph hint:* directives is to strip them entirely.  

It is important to note that QLever's optimizer can be influenced by the structure of a SPARQL query. It is recommended to use subqueries to force early evaluation of a portion of a query. When enclosing a block inside a SPARQL subquery (SELECT ... WHERE { ... }), QLever will execute that block before joining its results with the remainder of the query.

== wikibase:decodeURI ==
wikibase:decodeUri reverses percent-encoding (for example, é → %C3%A9, comma → %2C, etc.) in a string. It is the explicit inverse of SPARQL 1.1's ENCODE_FOR_URI and turns IRIs such as Wikipedia/Commons sitelink URLs into human-readable strings. 

Basically, this is a function that is used in BINDs and projected variable clauses to create  human output. An exemplary Blazegraph query that translates a Russian Wikipedia sitelink to readable characters is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?sitelink ?decodedURI ?title WHERE {
  ?sitelink schema:about ?item ;
                schema:isPartOf &lt;https://ru.wikipedia.org/&gt; .
  BIND(wikibase:decodeUri(STR(?sitelink)) AS ?decodedURI)
 } LIMIT 10
&lt;/syntaxhighlight&gt;

The following illustrates one of the results:
* ?item: http://www.wikidata.org/entity/Q14760226 
* ?sitelink: https://ru.wikipedia.org/wiki/%D0%90%D0%B1%D1%83%D0%BB%D1%8C_%D0%A5%D0%B0%D0%B4%D0%B6%D0%B4%D0%B6%D0%B0%D0%B4%D0%B6_I
* ?decodedURI: https://ru.wikipedia.org/wiki/Абуль_Хаджджадж_I

Regrettably, there is no equivalent decodeURI functionality in SPARQL 1.1. It cannot be created since that would require iterating over a string to assemble &lt;i&gt;multi-byte UTF-8 sequences&lt;/i&gt; (as seen above, %C3%A9 becomes é). SPARQL does not support loops or byte-level operations, so a complete and correct decoder is not possible.

The only approach is to translate the URI after the SPARQL results are returned. Every programming language has a URL decode component. Using one of these languages (such as python or perl) on a Mac or Linux system, the command line instruction is:
:&lt;syntaxhighlight lang=&quot;text&quot;&gt;
python3 -c &quot;import sys,urllib.parse as u; print(u.unquote(sys.argv[1]))&quot; &quot;uri_string_to_decode&quot;
&lt;/syntaxhighlight&gt;
OR
:&lt;syntaxhighlight lang=&quot;text&quot;&gt;
perl -CS -MURI::Escape -MEncode -e 'print decode_utf8(uri_unescape($ARGV[0])),&quot;\n&quot;' &quot;uri_string_to_decode&quot;
&lt;/syntaxhighlight&gt;

On Windows, it can be accomplished in Powershell by:
:&lt;syntaxhighlight lang=&quot;shell-session&quot;&gt;
pwsh -c &quot;[Uri]::UnescapeDataString('uri_string_to_decode')&quot;
&lt;/syntaxhighlight&gt;

An alternate strategy when decoding Wikipedia sitelinks is to instead retrieve the “title” from the value of the predicate, schema:name. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?title WHERE {
  ?article schema:about ?item ;
           schema:isPartOf &lt;https://ru.wikipedia.org/&gt; ;
           schema:name ?title .
} LIMIT 10 
&lt;/syntaxhighlight&gt;

Note that this does not return the full URI, but only the last path segment. So, at best, it is an approximation to aid in human readability.

=== Query Rewrite Pattern for wikibase:decodeURI ===
There is no general rewrite pattern. decodeURI() should be handled manually, at the client, using the command line instructions shown above. 

Specifically for sitelinks (identified as the subject variable of a triple with the predicate, schema:about), a decoded subset of the full URI could be obtained by retrieving the value of the variable’s schema:name predicate. 

&lt;i&gt;?something schema:about ?item … BIND(wikibase:decodeUri(STR(?something)) AS ?decodedURI)&lt;/i&gt; becomes &lt;i&gt;?something schema:about ?item ; schema:name ?decodedTitle&lt;/i&gt;. However, further string manipulations of ?decodedURI will require updating since ?decodedURI and ?decodedTitle are not equivalent. 

== wikibase:label ==
The wikibase:label service auto-resolves an entity's label, description, and aliases into result variables, honoring a language priority list with fallback. There are two modes used to define variable names - a manual and an automatic mode.

In the manual mode, the query writer declares the variable names. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   SERVICE wikibase:label {
       bd:serviceParam wikibase:language &quot;de,en&quot; .
       ?item rdfs:label ?label .
       ?item schema:description ?description .
       ?item skos:altLabel ?alternate . }
}
&lt;/syntaxhighlight&gt;

This query produces results as shown below:
[[File:Rewrite Label-Fig 1.png|Figure 1]]

In automatic mode, no names are defined and the service projects specific names for any variables, ?x. The projected names are ?xLabel, ?xDescription, ?xAltLabel. A query could be written as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel ?itemDescription ?itemAltLabel WHERE {
   ?item wdt:P31 wd:Q515 .
   SERVICE wikibase:label { 
       bd:serviceParam wikibase:language &quot;[AUTO_LANGUAGE],de,en&quot;. }
}
&lt;/syntaxhighlight&gt;

This query produces the same results but with different names for the label, description and alias outputs.

When rewriting a query to produce equivalent results, the rewrite must consider:
* Language priority + fallback — wikibase:language is a comma-separated priority list where the first available language wins
** Usage of [AUTO_LANGUAGE] will be continued as a UI-only feature (it expands the list of languages to include the requester's UI/browser language)
* mul fallback — a label or alias under the multilingual code “mul” applies to all languages and is consulted after the listed languages
** There are no “mul” descriptions 
* Allowance for unbound labels, descriptions and aliases
* QID fallback for labels (only)
** If no label exists in any requested language or mul, the entity ID (e.g., Q42) is returned 
** There is no fallback for descriptions or altLabels (they are unbound if not found)
* skos:altLabel can be multivalued
** label service returns all aliases of the best language as one comma-joined string
* Applicable to both items and properties 

At its most basic, the rewrite must drop the service call, determine each entity variable and its related label/description/alias variable names, and then return those variable values based on SPARQL OPTIONAL clauses. 

Why SPARQL OPTIONALs? Because defining a raw triple pattern (such as ?item schema:description ?description) without OPTIONAL requires the item to have a description. If one is not defined, then the entity will be deleted from the result set. This is not the behavior of the wikibase:label service.

To rewrite the first query in this section, a naive approach (but one that illustrates what is happening “under the covers”) produces the following query:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   # --- label: priority de &gt; en &gt; mul, else QID ---
   OPTIONAL { ?item rdfs:label ?label_de  . FILTER(LANG(?label_de)  = &quot;de&quot;)  }
   OPTIONAL { ?item rdfs:label ?label_en  . FILTER(LANG(?label_en)  = &quot;en&quot;)  }
   OPTIONAL { ?item rdfs:label ?label_mul . FILTER(LANG(?label_mul) = &quot;mul&quot;) }
   BIND(COALESCE(STR(?label_de), STR(?label_en), STR(?label_mul),
                 STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?label)
   # --- description: priority de &gt; en &gt; mul, else unbound ---
   OPTIONAL { ?item schema:description ?desc_de  . FILTER(LANG(?desc_de)  = &quot;de&quot;)  }
   OPTIONAL { ?item schema:description ?desc_en  . FILTER(LANG(?desc_en)  = &quot;en&quot;)  }     
   BIND(COALESCE(STR(?desc_de), STR(?desc_en)) AS ?description)
   # --- aliases: priority de &gt; en, multi-valued → GROUP_CONCAT per language ---
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_de) WHERE {
         ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;de&quot;)
      } GROUP BY ?item
   }
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_en) WHERE {
        ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;en&quot;)
      } GROUP BY ?item
   }
   OPTIONAL {
      SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alt_mul) WHERE {
        ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;mul&quot;)
      } GROUP BY ?item
   }
   BIND(COALESCE(?alt_de, ?alt_en, ?alt_mul) AS ?alternate)
}
&lt;/syntaxhighlight&gt;

There are several important things to note:
* All the resulting label/description labels are COALESCE’d as STR(?x) in order to remove the language tags
** This is needed since wikibase:label does not return language tags, but plain literals
** This is not needed for the aliases since GROUP_CONCAT returns a plain literal
* “mul” must be explicitly added as a language
* The COALESCE order is defined by the language priority with “mul” as the lowest priority
* The label’s COALESCE falls back to the entity QID if there are no labels in the specified languages
* Since aliases are multi-valued, each language’s results are GROUP_CONCAT'ed in their own GROUP BY ?item subqueries, then COALESCE'd for language priority
** There is a performance implication for this - the subqueries (SELECT clauses within the OPTIONALs) are always executed first and will concatenate alias names across all items with “de” or “en” aliases 
*** This is because there are no binding constraints on ?item
** To improve the performance, the constraining triple(s) for ?item (in this case, &lt;i&gt;?item wdt:P31 wd:Q515&lt;/i&gt;) should be moved inside the sub-SELECT scope

Executing the rewritten query obtains the following (equivalent but with different order) results:
[[File:Rewrite Label-Fig 2.png|Figure 2]]

The rewritten query was characterized as “naive” since it included the COALESCE. That forces an explicit ordering of the variable binding. However, it is not needed since the OPTIONAL clause is a left join. Once the first OPTIONAL binds ?label to the German label, the same variable in the next OPTIONAL's triple pattern is no longer a fresh binding. It becomes a join constraint: ?item rdfs:label ?label must now match the already-bound German literal, and FILTER(LANG(?label) = &quot;en&quot;) on a @de literal is false. Therefore, the “en” OPTIONAL adds nothing and the German value survives. If a German label did not exist (?label is unbound), the English OPTIONAL is free to bind it; then mul; and so on. The left joins themselves implement &quot;first match wins&quot; which is what COALESCE was doing (making it redundant).

What remains is only the QID fallback for ?label and stripping the language tag. That can be written without COALESCE:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
BIND(IF(BOUND(?label), STR(?label), STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?itemLabel)
&lt;/syntaxhighlight&gt;

Given this discussion, the rewritten query becomes:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label ?description ?alternate WHERE {
   ?item wdt:P31 wd:Q515 .
   # label: de &gt; en &gt; mul into one shared ?lbl (left joins give priority), else QID
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;de&quot;)  }
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;en&quot;)  }
   OPTIONAL { ?item rdfs:label ?lbl . FILTER(LANG(?lbl) = &quot;mul&quot;) }
   BIND(IF(BOUND(?lbl), STR(?lbl), STRAFTER(STR(?item), &quot;entity/&quot;)) AS ?label)
   # description: de &gt; en into shared ?dsc, no mul and no fallback; 
   # STR(?dsc) to remove lang tag
   OPTIONAL { ?item schema:description ?dsc . FILTER(LANG(?dsc) = &quot;de&quot;) }
   OPTIONAL { ?item schema:description ?dsc . FILTER(LANG(?dsc) = &quot;en&quot;) }
   BIND(STR(?dsc) AS ?description)
   # aliases: de &gt; en &gt; mul into shared ?alternate, constrained, GROUP_CONCAT
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;de&quot;)  
       } GROUP BY ?item }
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;en&quot;)  
       } GROUP BY ?item }
   OPTIONAL { 
       SELECT ?item (GROUP_CONCAT(DISTINCT ?a; SEPARATOR=&quot;, &quot;) AS ?alternate)
           WHERE { ?item wdt:P31 wd:Q515 . ?item skos:altLabel ?a . FILTER(LANG(?a) = &quot;mul&quot;) 
       } GROUP BY ?item }
}
&lt;/syntaxhighlight&gt;

The rewrite for the second query in this section (using automatic variable name binding and AUTO-LANGUAGE) uses the same overall pattern. But, there are a few items to note:
* The AUTO_LANGUAGE UI token will be maintained - so, it can be substituted as a language directly in the rewritten query (e.g., LANG(?x) = “AUTO_LANGUAGE”)
** Outside of the UI, there is no equivalent concept - all languages must be explicitly declared in the query
* If the query projected additional entity variables beyond ?item (e.g. ?country), each would need its own blocks for determining ?countryLabel, ?countryDescription and ?countryAlias 

=== Query Rewrite Pattern for wikibase:label ===
Collecting all the requirements and considerations above, the rewrite pattern can be explained as:
* Determine the languages to be output, splitting wikibase:language values using a comma delimiter
** And their priority order
* Determine the names of the projected variables 
** Which then defines the types of OPTIONAL “blocks” needed - label/description/alias
* Create those blocks based on the pattern shown in the last query above, and substitute them in the query in place of the wikibase:label service
** Taking care to re-insert the constraining triple(s) for the variable in the alias block

== Footnotes ==</text>
      <sha1>9w0q75leehs373tax0zekr8bjjxqjjv</sha1>
    </revision>
  </page>
  <page>
    <title>Blazegraph Migration: Rewrite of MWAPI</title>
    <ns>0</ns>
    <id>460377</id>
    <revision>
      <id>2433188</id>
      <parentid>2431228</parentid>
      <timestamp>2026-07-05T02:50:31Z</timestamp>
      <contributor>
        <username>Quiddity</username>
        <id>1884</id>
      </contributor>
      <comment>lang=&quot;sparql&quot;; wikify</comment>
      <origin>2433188</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="23264" sha1="1vy4cg6n6t39ft07l3mslc7vl9nuaoe" xml:space="preserve">MWAPI is one aspect of search on Blazegraph. Another is Blazegraph’s search (bds:) predicates, which exist in the Wikidata fork.

But, this page is focused on MWAPI. Why? bds:search requires a full-text (Lucene) index to be built over all the literals. That is not done for Wikidata. Without the index, the predicates (bds:search and its modifying predicates, bds:matchAllTerms, bds:relevance, bds:min/maxRelevance, bds:rank and bds:min/maxRank) either return nothing or error out. So, instead of bds:search, there is MWAPI.

[https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI SERVICE wikibase:mwapi { ... }] is a Blazegraph-specific extension to WDQS that
calls the MediaWiki Action API and binds its results to SPARQL variables. The service is not part of the SPARQL 1.1 standard since it is not possible to call a REST API within a query. 

MWAPI’s bridges the MediaWiki Action API into SPARQL - adding full-text search via CirrusSearch (backed by an Elasticsearch/OpenSearch cluster) over any Wikimedia project including Wikidata, label/alias lookup via wbsearchentities, and structural queries (categories, backlinks, transclusions) via generators.

The following table briefly highlights the different searches possible using the MWAPI service on WDQS and the corresponding “actions” if using the MWAPI directly:
{| class=&quot;wikitable&quot;
|-
! Service !! MWAPI Action !! Inputs !! Outputs !! Description
|-
| Generator || action=query&amp; generator=&lt;name&gt; || generator, prop, pprop || title, item, pageid, lastrevid, timestamp || Generators retrieve a base set of pages or items. The property modules fetch specific data for the retrieved items.
|-
| Categories || action=query&amp; prop=categories || titles, cllimit || category, title || Lists the categories that a given page belongs to.
|-
| Search || action=query&amp; list=search || srsearch, srwhat, srlimit || title || Provides full-text CirrusSearch on the target wiki.
|-
| EntitySearch || action= wbsearchentities || search, language, type, limit || item, label || Performs Wikidata entity label and alias search.
|}

Overall, this is a huge topic and MWAPI’s functionality cannot be replicated using standard SPARQL 1.1 syntax. This is because the behavior depends on relevance ranking, case and accent folding, and reliance on data outside Wikidata. 

The scenarios where rewrites are theoretically possible (although not equivalent) are:
* EntitySearch/wbsearchentities requests
* Search/CirrusSearch addressed to www.wikidata.org

These specific scenarios are briefly discussed in later portions of this document. But, there is no way to achieve full equivalency. 

Note that the ''only reliable and efficient rewrite'' is to separately call the MediaWiki Action API and then feed the results into a subsequent SPARQL query (which is discussed in the next section). 

== Using the MediaWiki Action API ==
This section describes the &quot;two-step pattern&quot; (Action API -&gt; SPARQL query) referenced above. It performs the search/generator step using the MediaWiki endpoint (https://www.wikipedia.org/w/api.php), collects the Wikidata entities (QIDs) and then uses these in a SPARQL 1.1 query with a VALUES block.  

The pattern can be accomplished using `curl` plus `jq`&lt;ref&gt; jq is a lightweight, command-line tool to process, filter and extract data from JSON. See https://jqlang.org/.&lt;/ref&gt; via the command line, or using a programming language such as Python. The Python code is also included below, since it can implement robust pagination (continuation processing) for bulk data handling. Also, setting the properties of the API call is easier than manipulating the command line.

=== Requirements and Constraints of Using the API ===
* Mandatory (defined by Wikimedia policy):
** Set a `User-Agent` with a contact URL/email 
*** Missing or generic UAs will immediately fail with an authorization (403) error 
* Correctness (as specified by the API contract):
** Use `format=json&amp;formatversion=2`
** Never assume that a request is complete
*** MWAPI may return a continue element in the JSON response
*** Clients must append a sub-element of “continue” as a parameter to subsequent requests, to fetch the next batch of data
* Per-call limits for MWAPI with WDQS: 
** wbsearchentities: 7 (default for web clients and regular users); 50 (API and bot users); Max 50
** Search: 10 (default); Max 500 (standard users); Max 5,000 (bots); Hard-coded max 10000 rows (with use of continuation and `sroffset`)
* Robustness (client-side):
** Always check for an `error` key before using the query results
* Etiquette/Being a good-citizen (recommended for bulk processing):
** Timeout handling
** `maxlag=5` + backoff on a `maxlag` error

=== Use of curl and jq ===
The following should be sufficient for a one-off or a small result set. This approach has no dependencies beyond `curl` and `jq`:

:1. Set the user-agent
::* For example, ''UA='mwapi-substitute/0.5 (User: awesterinen-ctr@wikimedia.org)'''
::* For more information, see our [https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy user-agent policy] and [https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_API_Usage_Guidelines API usage guidelines]
:2. Utilize the Action API to search Wikidata for specific entities (QIDs), then use jq to parse the returned JSON to extract them. Lastly, the results are redirected to a file for persistence/use in WDQS calls.
::*For example, to get all QIDs whose “en”/”mul” labels or aliases include “Einstein”,  the following command is executed: 
:::''&lt;nowiki&gt;curl -s -G 'https://www.wikidata.org/w/api.php' -H &quot;User-Agent: $UA&quot; --data-urlencode 'action=wbsearchentities' --data-urlencode 'search=Einstein' --data-urlencode 'language=en' --data-urlencode 'type=item' --data-urlencode 'format=json' --data-urlencode 'formatversion=2' | jq -r '.search[].id' &gt; qids.txt&lt;/nowiki&gt;''
::*There are 7 results returned&lt;ref&gt;7 is the default number returned when a limit is not specified.&lt;/ref&gt;: Q16834800, Q937, Q901448, Q1309274, Q26854017, Q1892, Q28739649
::* Note that the full JSON response to the wbsearchentities request contains much more information than just the QIDs:
:::[[File:MWAPI Rewrite-Fig 1.png|Figure 1]]
:3. Build a VALUES string from the QIDs, to use in the SPARQL query
::* For example, ''VALUES=$(paste -sd' ' &lt;(sed 's/^/wd:/' qids.txt))'' 
::* Which results in the string, “wd:Q16834800 wd:Q937 wd:Q901448 wd:Q1309274 wd:Q26854017 wd:Q1892 wd:Q28739649”
:4. Include the results in a request to WDQS
::* For example, 
:::''&lt;nowiki&gt;curl -s -G 'https://query.wikidata.org/sparql' -H &quot;User-Agent: $UA&quot; -H 'Accept: application/sparql-results+json' --data-urlencode &quot;query=PREFIX wd: &lt;http://www.wikidata.org/entity/&gt; PREFIX rdfs:&lt;http://www.w3.org/2000/01/rdf-schema#&gt; SELECT ?item ?label WHERE { VALUES ?item { $VALUES } ?item rdfs:label ?label . FILTER(LANG(?label)='en') }&quot;&lt;/nowiki&gt;''
::* Or as input to the WDQS UI, as shown below
:::&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?label WHERE {
   VALUES ?item { wd:Q16834800 wd:Q937 wd:Q901448 wd:Q1309274
                  wd:Q26854017 wd:Q1892 wd:Q28739649 }
   ?item rdfs:label ?label. FILTER(LANG(?label) = &quot;en&quot;)
}
&lt;/syntaxhighlight&gt;
::* Which returns the results:
:::[[File:MWAPI Rewrite-Fig 3.png|Figure 3]]

Using the command line is awkward exactly where Python is useful. Python allows for handling continuations, errors or mandatory lags, or chunking a large number of VALUES.

=== Use of Python for Pagination, Chunking and Error Handling ===
The code in the Jupyter notebook, [https://github.com/AndreaWesterinen/Wikidata-Queries/blob/main/using_mwapi/MWAPI.ipynb MWAPI.ipynb], implements the pattern from above, and adds support for larger limits, continuation, CirrusSearch and generator processing.

There are examples of using MWAPI for Wikidata label/alias search (similar to what is shown for `curl` above), CirrusSearch and several generators. Note that the generators used in the notebook return Wikipedia articles. The Python code converts those page references to Wikidata QIDs as part of the ''generator-to-qids'' function.

After obtaining the QIDs, these can then be fed into SPARQL 1.1 queries (using the VALUES keyword). The code to perform the queries is also included in the second to last cell of the notebook (as a query with a very minimal result set - basically, returning the “instance of” entities for the QID, and the labels of the QID and the “instance of” entity). Also, a Python-only “pretty print” routine is shown in the notebook, in the last cell.

An example of the output for the wbsearchentities results is shown below:
:{| class=&quot;wikitable&quot;
|-
! item !! type !! label !! typeLabel 
|-
| Q136164891 || Q3331189 || Einstein || version, edition or translation
|-
| Q11452 || Q214070 || general relativity || physical law 
|-
| Q59151 || Q173227 || cosmological constant || physical constant 
|-
| Q138088407 || Q1542966 || Einstein-Gymnasium Neuenhagen || gymnasium 
|-
| Q135050506 || Q7889 || Einstein's Cats || video game 
|-
| Q17712 || Q131647 || Albert Einstein Medal || medallion 
|-
| Q35875 || Q24034552 || mass–energy equivalence || mathematical concept 
|-
| Q131938472 || Q11707 || Einstein - Aarau || restaurant  
|-
| Q131938473 || Q11707 || Einstein Bistro - St. Gallen || restaurant 
|-
| Q35875 || Q33104303 || mass–energy equivalence || concept in physics  
|-
| Q1892 || Q11344 || einsteinium || chemical element   
|}

== Qualified Query Rewrites for MWAPI ==
The section above explained how to generically call MWAPI and transform/use the results. However, there are some specific MWAPI queries that appear to be possible to transform to SPARQL 1.1 compliant rewrites. These are discussed below, along with explaining their lack of equivalency.

=== EntitySearch/wbsearchentities ===
The MWAPI service's ''EntitySearch'' template (a wrapper for the wbsearchentities Action API) performs a start-anchored, language-aware, relevance match over Wikidata items’ labels and aliases. It returns the top-ranked entities whose label or alias ''begins'' with the specified search string, in the selected and fallback languages.

A similar intent can be expressed in plain SPARQL 1.1 by adding the following to a query that binds the ?item variable. For example:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
?item rdfs:label|skos:altLabel ?match .
FILTER(LANG(?match) IN (&quot;en&quot;, &quot;mul&quot;))
FILTER(STRSTARTS(LCASE(STR(?match)), &quot;some_text&quot;))
&lt;/syntaxhighlight&gt;

Unfortunately, there is no string index behind STRSTARTS nor is there any support for accent folding&lt;ref&gt;A text preprocessing technique that strips accent marks and diacritics&lt;/ref&gt; or relevance scoring. The SPARQL engine evaluates the specified filter on every label or alias for the items that are bound in the query. If the query sufficiently constrains ?item (such as a structural constraint like ''?item wdt:P31 wd:&lt;TYPE&gt;'' where there are less than 2M possible items), the query may execute within the timeout period. However, if there are no bounds or their evaluation results in many millions of possible ?items, the query will certainly timeout. 

As an example, here is a MWAPI EntitySearch query (and its results) for entities with labels/aliases starting with “saint”. Note that there is no criteria restricting the kinds of entities returned.
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
  SELECT ?item ?label WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api &quot;EntitySearch&quot; ;
                    wikibase:endpoint &quot;www.wikidata.org&quot; ;
                    mwapi:search &quot;saint&quot; ;
                    mwapi:language &quot;en&quot; .
    ?item  wikibase:apiOutputItem mwapi:item .
    ?label wikibase:apiOutput  mwapi:label .
  }
}
&lt;/syntaxhighlight&gt;

The following top-ranked&lt;ref&gt;‘top-ranked’ is determined by sitelink count, statement count, matching the label versus the alias, matching the specified language versus the fallback, etc. This ordering cannot be replicated in SPARQL. Also, note that 50 is the limit for the number of items returned by EntitySearch/wbsearchentities.&lt;/ref&gt; results are reported:
:[[File:MWAPI Rewrite-Fig 5.png|Figure 5]]

Executing an equivalent query using SPARQL 1.1 would certainly time out since all Wikidata entities’ labels/aliases would be FILTERed.

Further complicating matters, another difference in the results occurs when different spellings, accents and diacritics must be taken into account. As noted earlier, EntitySearch/wbsearchentities results are accent folded. This means that the string “Curie” matches “Curié”, and the string “Lodz” matches “Łódź”. To do this sort of filtering with SPARQL would require rewriting the filter as: 
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
?item rdfs:label|skos:altLabel ?match .
FILTER(LANG(?match) IN (&quot;en&quot;, &quot;mul&quot;))
BIND(LCASE(?match) AS ?lc)
FILTER(STRSTARTS(LCASE(STR(?lc)), &quot;some_text&quot;) || 
             STRSTARTS(LCASE(STR(?lc)), &quot;some_alternate_version_of_text&quot;) || 
             … )
&lt;/syntaxhighlight&gt;

The above would be difficult to correctly and fully state.

In this case, the only resort is to separately call the MWAPI, and then use the results in a separate query.

=== CirrusSearch of Wikidata ===
Every reason to discourage rewriting EntitySearch/wbsearchentities applies to Search/CirrusSearch as well, plus several additional reasons exist that are unique to the broader feature set which CirrusSearch exposes.

The shared reasons are: 
* No case folding in SPARQL → Need to FILTER LCASE
* No accent folding → Need to FILTER on multiple patterns, or use some REPLACE/REGEX patterns 
* No language fallback → Need explicit LANG IN (&quot;en&quot;, &quot;mul&quot;, …) statements
* No relevance ranking → SPARQL has no concept of “top-ranked” and so can return large result (or intermediate result) sets
* Unindexed-scan ceiling → As above, can easily time out for queries that have no bounding/constraining criteria

Additional reasons for lack of equivalency in SPARQL 1.1 are:
* Tokenization includes word-boundary semantics → For example, ''inlabel: theater'' matches &quot;Dream Theater&quot;; This could be accomplished using SPARQL’s CONTAINS keyword but that could return unwanted results
** For example, consider “theatergoer”: MWAPI would not match this string since it would not match a word boundary, but SPARQL CONTAINS would indicate a match
* Stemming → For example, “theory” matches &quot;theorical&quot; in CirrusSearch whereas SPARQL CONTAINS would not
* morelike:QID → Finds items considered “similar” to the specified one; There is no equivalent SPARQL rendering 
* Fuzzy matching (via a trailing ~) → Supports Damerau-Levenshtein edit distance which has no corollary in SPARQL 
* Regex with stem/case awareness → CirrusSearch's inlabel:/.../ does full-string matching, (it requires the pattern to consume the ''entire'' field value), whereas SPARQL REGEX does substring matching (the pattern just has to appear somewhere in the literal) 
** For example, consider the labels, “Theater”, “Dream Theater” and “Theatergoer”
** CirrusSearch’s regex might search for ''inlabel:/theater/i'', and would only match “Theater”
** SPARQL's REGEX equivalent must be written as ''FILTER(REGEX(?label, &quot;^theater$&quot;, &quot;i&quot;))'' to force full-string matching
* Text search across labels, aliases, descriptions, and the labels of statement values in one pass → Can be reproduced in SPARQL with explicit UNION over rdfs:label, skos:altLabel, schema:description, and joined statement-value labels

There is a limited case where rewriting to SPARQL 1.1 is possible - when the CirrusSearch query (against the Wikidata endpoint) uses the haswbstatement structural keyword with (optionally) haslabel: or hasdescription:.

For example, consider the case of searching for female astronauts:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api &quot;Search&quot; ;
             wikibase:endpoint &quot;www.wikidata.org&quot; ;
             mwapi:srsearch &quot;haswbstatement:P106=Q11631 haswbstatement:P21=Q6581072&quot; ;                                
             mwapi:srlimit &quot;max&quot; .
   ?bare_qid wikibase:apiOutput mwapi:title .   
  }
  BIND(IRI(CONCAT(STR(wd:), ?bare_qid)) AS ?item)
  SERVICE wikibase:label { bd:serviceParam wikibase:language &quot;en&quot;. }
}
&lt;/syntaxhighlight&gt;

Before looking at the results, it is important to examine the query. Executing MWAPI “Search” against the Wikidata endpoint returns a single result - mwapi:title. But, it only returns that result when specifically requested as the apiOutput - as is done in line 7. The “title” that is returned is a Wikidata QID - but not an IRI, just the string, “Q###”. To use this result in subsequent triples in the query, it must be converted to an IRI. That is accomplished in line 9.

The above query returns results as shown below:
[[File:MWAPI Rewrite-Fig 6.png|Figure 6]]

To execute the equivalent SPARQL 1.1 query, it must be expressed as:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT * WHERE { 
   ?item p:P106/ps:P106 wd:Q11631 ;
             p:P21/ps:P21 wd:Q6581072 .                                
   OPTIONAL { ?item rdfs:label ?label . FILTER(LANG(?label) = &quot;en&quot;) }                  
   OPTIONAL { ?item rdfs:label ?label . FILTER(LANG(?label) = &quot;mul&quot;) }
}
&lt;/syntaxhighlight&gt;

Why? Because haswbstatement is rank-blind. It can return deprecated statements. These must be accounted for in the rewritten SPARQL by examining the qualified properties/statements.

It is valuable to note that the SPARQL query executes in less time (on both Blazegraph and QLever) than the corresponding MWAPI request.

As another example, consider the example of searching for female astronauts that also have a label in German. Using the MWAPI, the query is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel WHERE {
 SERVICE wikibase:mwapi {
   bd:serviceParam wikibase:api &quot;Search&quot; ;
            wikibase:endpoint &quot;www.wikidata.org&quot; ;
            mwapi:srsearch &quot;haswbstatement:P106=Q11631 haswbstatement:P21=Q6581072 
                            haslabel:de&quot; ;                                
            mwapi:srlimit &quot;max&quot; .
   ?bare_qid wikibase:apiOutput mwapi:title .   
 }
 BIND(IRI(CONCAT(STR(wd:), ?bare_qid)) AS ?item)
 SERVICE wikibase:label { bd:serviceParam wikibase:language &quot;en&quot;. }
}
&lt;/syntaxhighlight&gt;

125 results are returned. 

The equivalent (and faster) SPARQL 1.1 query is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel WHERE { 
   ?item p:P106/ps:P106 wd:Q11631 ;
             p:P21/ps:P21 wd:Q6581072 .       
   ?item rdfs:label ?germanLabel . FILTER(LANG(?germanLabel) = &quot;de&quot;) 
   OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?label) = &quot;en&quot;) }                  
   OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?label) = &quot;mul&quot;) }
}
&lt;/syntaxhighlight&gt;

== Query Rewrite Pattern ==
'''The only cleanly rewritable MWAPI queries use CirrusSearch of Wikidata and the keywords: haswbstatement:, haslabel:, and hasdescription:. They can include boolean composition operators (AND/OR/-) and use the syntax, [property_PID=entity_QID], for searching qualified statements.''' 

The rewrite pattern follows this algorithm:
* Parse the CirrusSearch query into clauses by:
** Splitting by parentheses
** Examining the contents of each:
*** Recurse if there are multiple levels of parentheses to reach the inner clauses
*** Split on the top-level boolean operators: whitespace AND and OR
*** For each leaf:
**** Strip a leading - (negation) if present (it doesn't change whether the search is rewritable or not)
**** Make sure that the leaf starts with one of: haswbstatement:, haslabel:, or hasdescription:
***** If not, then the query cannot be rewritten and requires an external call to MWAPI and subsequent use of the results in a SPARQL query
* Apply per-clause rewrite rules:
** haswbstatement:P##=Q## → ''?item p:P##/ps:P## wd:Q## .''
** hswbstatement:P##=Q##[P**=Q**] → ''?item p:P## ?statement. ?statement ps:P## wd:Q## ; pq:P** wd:Q** .''  
** haslabel:LANG → ''FILTER EXISTS { ?item rdfs:label ?l . FILTER(LANG(?l)=&quot;LANG&quot;) }''
** hasdescription:LANG → ''FILTER EXISTS { ?item schema:description ?d . FILTER(LANG(?d)=&quot;LANG&quot;) }''
** - (negation) → ''FILTER NOT EXISTS { body_rewritten_using_rules }''
* Recreate the boolean structure 
** AND → Multiple triple patterns / filters in the same WHERE block
** OR → Wrap the alternatives in { … } UNION { … } and add DISTINCT to the outer SELECT to dedupe
** Grouping ( … ) → Preserved by { … } blocks around the rewritten body
* Add ''LIMIT 500'' to approximate the Search “standard user” cap

Note that even this rewrite will not be equivalent since CirrusSearch returns the results ranked by relevance, and SPARQL returns the entire set of results.

Examining a specific example, consider the CirrusSearch query for female astronauts without a French language description:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api &quot;Search&quot; ;
                    wikibase:endpoint &quot;www.wikidata.org&quot; ;
                    mwapi:srsearch &quot;haswbstatement:P106=Q11631 
                                               haswbstatement:P21=Q6581072 -hasdescription:fr&quot; ;
                    mwapi:srlimit &quot;max&quot; .
    ?title wikibase:apiOutput mwapi:title .
  }
  BIND(IRI(CONCAT(STR(wd:), ?title)) AS ?item)
  SERVICE wikibase:label { bd:serviceParam wikibase:language &quot;en&quot;. }
}
&lt;/syntaxhighlight&gt;

This produces 52 results. A portion of the output is shown:
[[File:MWAPI Rewrite-Fig 7.png|Figure 7]]

The rewritten query is:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT DISTINCT ?item ?itemLabel WHERE {
    ?item p:P106/ps:P106 wd:Q11631 ;       # rank-blind: astronaut
              p:P21 /ps:P21  wd:Q6581072 .     # rank-blind: female
  FILTER NOT EXISTS {                            # negated hasdescription:fr
     ?item schema:description ?d . FILTER(LANG(?d) = &quot;fr&quot;) }
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = &quot;en&quot;) }
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = &quot;mul&quot;) }
} LIMIT 500
&lt;/syntaxhighlight&gt;

Which also produces 52 results.

Alternately, consider the query for people who received a PhD from MIT:
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT ?item ?itemLabel WHERE {
  SERVICE wikibase:mwapi {
    bd:serviceParam wikibase:api &quot;Search&quot; ;
                    wikibase:endpoint &quot;www.wikidata.org&quot; ;
                    mwapi:srsearch &quot;haswbstatement:P69=Q49108[P512=Q849697]&quot; ;
                    mwapi:srlimit &quot;max&quot; .
    ?title wikibase:apiOutput mwapi:title .
  }
  BIND(IRI(CONCAT(STR(wd:), ?title)) AS ?item)
  SERVICE wikibase:label { bd:serviceParam wikibase:language &quot;en&quot;. }
} 
&lt;/syntaxhighlight&gt;

Which returns 67 results.

This is translated to the SPARQL query (which also produces 67 results):
&lt;syntaxhighlight lang=&quot;sparql&quot;&gt;
SELECT DISTINCT ?item ?itemLabel WHERE {
  ?item p:P69 ?s . ?s ps:P69 wd:Q49108 ; pq:P512 wd:Q849697 .
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = &quot;en&quot;) }
  OPTIONAL { ?item rdfs:label ?itemLabel . FILTER(LANG(?itemLabel) = &quot;mul&quot;) }
} LIMIT 500
&lt;/syntaxhighlight&gt;

== Footnotes ==</text>
      <sha1>1vy4cg6n6t39ft07l3mslc7vl9nuaoe</sha1>
    </revision>
  </page>
</mediawiki>